< midnightmagic>
getting bitcoin built on 10.04 is turning into quite the chore.
< phantomcircuit>
midnightmagic, why are you trying to build on 10.04?
< midnightmagic>
phantomcircuit: so I don't have to completely rebuild that machine from scratch, since modern kernels do not seem to support some of the hardware on that machine (including iwl4965agn)
< phantomcircuit>
you sure i have the driver for that on this debian system
< phantomcircuit>
midnightmagic, the 4965agn requires a binary blob firmware
< midnightmagic>
phantomcircuit: high-throughput usage in my wifi network freezes the card/driver/whatever and pauses all effective transmission. It must be full-duplex communication, so, tcp stream e.g., and not just a plain pingflood.
< midnightmagic>
phantomcircuit: yes, that it does. :)
< * midnightmagic>
does triple-take at phantomcircuit talking
< bitcoin-git>
[bitcoin] anditto opened pull request #9407: Added missing colons in when running help command (master...minor-style-fixes) https://github.com/bitcoin/bitcoin/pull/9407
< btcdrak>
wumpus: what's the plan for 0.13.2 - I assume we dont need another RC for the version number thing.
< gmaxwell>
btcdrak: still three days to christmas, plenty of time for more RCs.
< rabidus_>
:D
< btcdrak>
"on the day of Christmas, my true love gave to me, a RC in a merkle tree"
< gmaxwell>
man I ran 0.13.2(rc) in valgrind overnight last night on my laptop (which was a day or so behind) and it only synced 111 blocks in about 8 hours.
< midnightmagic>
time to write a custom memory manager
< bitcoin-git>
bitcoin/master afe5b3f Anditto Heristyo: Added missing colons in when running help command
< bitcoin-git>
bitcoin/master 041331e MarcoFalke: Merge #9407: [Trivial] Added missing colons in when running help command...
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #9407: [Trivial] Added missing colons in when running help command (master...minor-style-fixes) https://github.com/bitcoin/bitcoin/pull/9407
< jonasschnelli>
gmaxwell: "synced 111 blocks in about 8 hours" ... hah!
< jonasschnelli>
Whats the result? Any major leaks?
< gmaxwell>
no, course not.. bitcoind is leak free (ignoring the one shot openssl stupidity), I run every release in valgrind and have for a long time. (QT I wasn't doing that.)
< gmaxwell>
valgrind's leak detection is really not the interesting feature.
< gmaxwell>
the inderesting feature is uninitilized memory access detection.
< gmaxwell>
it's just become increasingly slow, I guess due to transaction traffic growth. :(
< luke-jr>
LLVM's thing is somewhat faster, but.. a hassle to use and broken with some LLVM/kernel pairs :/
< wumpus>
btcdrak: that's mostly an academic concern, how large is the chance that no issues come up with rc1 :) but if none happen, yeah I tend to agree, version bump can be part of final
< wumpus>
btw what about the meetings around xmas? I won't be able to attend this week's and next one - should I cancel them or just find another chair?
< gmaxwell>
You mean today's? for this weeks'? well we can always chat with whomever shows up, meeting or not.
< wumpus>
yes
< gmaxwell>
Hotel Ircfornia. You can check-out any time you like but you can never leave.
< wumpus>
but some people may be making time or getting up early for it or I dunno, if almost no one turns up that may be wasted effort
< gmaxwell>
I'll be around.
< wumpus>
ok, I guess you can be chair then :)
< gmaxwell>
k
< morcos>
instagibbs: heard you were thinking about allowing multiple outstanding block requests...
< morcos>
havent thought through yet how it might be implemented
< morcos>
i'm not sure that's 100% clear but there are a couple of different changes there
< morcos>
i think its kind of silly that we ignore cmpctblocks we receive from HB peers (if they don't immediately reconstruct) if we have an outstanding request for a cmpctblock from a LB peer.
< instagibbs>
trying to parse it now, it seems related to what I'm trying
< instagibbs>
Right now all I've attempted is to reply to a CMPCTBLOCK while the same block is in flight from another peer, responding up to 1 non-marked-as-hb peer, and responding to any marked-as-hb peer.
< instagibbs>
meaning if marked-as-hb is first, i wont respond to not-marked-as-hb peer
< morcos>
but you will respond to multiple hb peers?
< morcos>
i do think that if the first peer that sends you something is an HB peer, you should still request a cmpct block from a LB peer if they are the next one to inform you (via header)
< instagibbs>
Well, as-is yes, but I can make this "no" just as easily
< morcos>
basically i think if you consider a getdata BLOCKTXN or getdata BLOCK a second stage request, and a getdata CMPCTBLOCK a first stage request
< morcos>
you should be willing to always make sure you have at least two requests at as a high stage as you have opportunity for (except not 2 full BLOCK)
< instagibbs>
high stage? first? second? sorry this terminology is putting me in loops
< morcos>
eh, everytime i try to describe it even i don't think its clear
< instagibbs>
yeah...
< morcos>
i'll try another approach
< instagibbs>
Might be easier to talk about motivation, what we're gaining and avoiding
< morcos>
sorry i'm trying to build a spreadsheet, ha
< instagibbs>
oh jeez :)
< morcos>
my goal is to make it so that no one slow peer can stop one fast peer from getting the block to you as quickly as possible
< morcos>
as long as that fast peer is at least a compact block type (but he might be LB)
< instagibbs>
Agreed, my patch supposedly does this
< dgenr8>
morcos: does core try to ensure somehow that incoming txes paying the wallet are not dumped by mempool limiting?
< gmaxwell>
dgenr8: no doing so would be bad for privacy and reliablity (making the wallet think transactions were likely in other peoples mempools when they are not), and isn't needed.
< jonasschnelli>
gmaxwell: it's meetingstart I guess
< jl2012>
hi
< gmaxwell>
#meetingstart
< jonasschnelli>
well...
< CodeShark>
I just have one item on my agenda for today's meeting, but we don't have to go over it first
< gmaxwell>
hah the bot isn't here.
< instagibbs>
here
< gmaxwell>
well we don't need it. We can pretend its here.
< jonasschnelli>
Lightningbot has already xmas break,
< jonasschnelli>
sure
< gmaxwell>
In any case wumpus is out today, I'm sure many people are.
< gmaxwell>
Suggested topics?
< CodeShark>
WitnessMerkleBlock
< jonasschnelli>
If priorities allows, hd chain split and pub-key-der
< btcdrak>
lightningbot seems to be offline
< CodeShark>
we'll have to do it manually
< CodeShark>
unless there's a quick fix
< cfields>
sorry, here
< gmaxwell>
I presume everyone knows 0.13.2rc is out, and is busy testing it? :)
< instagibbs>
oh, no i didnt
< CodeShark>
+1 :)
< instagibbs>
#action test 0.13.2rc2
< instagibbs>
oh rc1 misread
< gmaxwell>
we tagged rc1, 0.13 branch is one PR ahead ... rc1 failed to bump the version. 0.13 branch does.
< gmaxwell>
Indeed:
< gmaxwell>
#action test 0.13.1rc1 or 0.13 branch
< gmaxwell>
(we're pretending the bot is here, remember.)
< jonasschnelli>
Yes. Testing here. Can't sign the gitian assets file right now,... key expired.
< gmaxwell>
jonasschnelli: you can always edit the expiration date on your pgp keys. the idea is to force others to get an updated copy of your key so they would have an oppturnity to learn of a revocation.
< jonasschnelli>
I really should create a new one.
< MarcoFalke_web>
?why
< jonasschnelli>
I try to use a HWW
< instagibbs>
gmaxwell, huh, TIL.
< gmaxwell>
(So it's a good practice to set a short expiration and then just keep periodically pushing it forward so long as your key isn't compromised)
< gmaxwell>
(e.g. short like a year and push it forward every six months)
< BlueMatt>
next topic? I know everyone has their pet feature for 0.14, given that we're getting close to feature-freeze already - but I wanted to nominate my pet for review-help :)
< CodeShark>
WitnessMerkleBlock :)
< instagibbs>
shoot blueman
< CodeShark>
but go for it, matt
< gmaxwell>
CodeShark: we'rell get to it!
< CodeShark>
:)
< jl2012>
topic suggestion: excessive sighashing protection policy #8755 and #8654. Maybe too late for 0.14?
< * BlueMatt>
wants to see parallel ProcessMessages - it seems like a lot to chew, but I think y'all might be surprised by the relatively simple patches to do it
< * CodeShark>
bites
< * BlueMatt>
did some work on running bitcoin core in helgrind, and finds it to be realatively simple
< BlueMatt>
but has opportunity to be a massive speedup in block-relay times
< jonasschnelli>
gmaxwell did
< gmaxwell>
jl2012: I don't know what wumpus is going to call for 0.14 though due to the net refactors I feel like we're going to have a relatively long finalization cycle for this release.
< BlueMatt>
instagibbs: mid-january feature freeze, actual freeze later
< gmaxwell>
I guess it would be good for people to propose 0.14 tags on whatever PRs are already in flight that you want in 0.14.
< cfields>
gmaxwell: i don't think it'll be too bad. Sadly it looks like the real async changes aren't going to make it for 0.14. I think the only real gotcha right now is the assert that keeps finding new issues, but we'll need to remove that soon
< btcdrak>
we should make a point to review 8755 and 8654.
< gribble>
https://github.com/bitcoin/bitcoin/issues/7533 | RPC: sendrawtransaction: Allow the user to ignore/override specific rejections by luke-jr · Pull Request #7533 · bitcoin/bitcoin · GitHub
< gmaxwell>
cfields: I'm disappointed with how ineffective testing is at finding issues behind that assert.
< cfields>
heh. let's take turns :)
< gmaxwell>
luke-jr: What do you think remains for multiwallet just more review? I admit I haven't been looking at it. I'd certantly like to see it in.
< gmaxwell>
luke-jr: I'm also reminded that you PRed a feature to sweep funds based on scanning the utxo set. Where is that right now?
< BlueMatt>
cfields: I'm done with my pet, just noting that to make it happen its gonna be a lot of prs that build on each other in quick succession so gonna need help with review there
< luke-jr>
gmaxwell: yes, pretty much; the other pre-multiwallet PR needs rebasing yet again, but that shouldn't be hard
< instagibbs>
I'm working on a couple CB PRs that would be nice to be in. But both are quite raw, maybe shoot for at least the first.
< luke-jr>
sweepprivkeys is #9152 but hasn't had any review yet (besides concept discussion of rawtx interfaces and such)
< gribble>
https://github.com/bitcoin/bitcoin/issues/9152 | Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local wallet by luke-jr · Pull Request #9152 · bitcoin/bitcoin · GitHub
< jonasschnelli>
IMO most important wallet features is the hd chain split
< jonasschnelli>
Better fix sooner then later
< luke-jr>
no objection to targetting 0.14 for it, but I don't consider it a huge priority since it is fairly isolated and easy to rebase
< jonasschnelli>
sweepprivkey should fit into a the rawtx line. I think this is too late for 0.14
< jonasschnelli>
Multiwallet probably also. This is an impactfull change.
< jonasschnelli>
I think it would be great if someone could review #9294
< gmaxwell>
jonasschnelli: Because of the lack of backwards compatiblity there can be an argument for batching it with other wallet changes. Though I agree it is really important.
< jonasschnelli>
I'd like to batch it with the HD-pubkey pr. But not sure if this is realistic.
< gmaxwell>
jonasschnelli: HD-pubkey pr?
< jonasschnelli>
IMO we should slowly think about a feature to use the wallet without privatekeys but with HD metadata.
< jonasschnelli>
Not sure if this is the right approach
< gmaxwell>
jonasschnelli: ah the change to not save private keys for the HD children but only derrive them when needed? yes, thats a wallet format change too.
< jonasschnelli>
But if we once like to support a process handling the private keys (like GPG agent) or compatibility with hardware wallets, we need something like this
< CodeShark>
it's the way to do it, ultimately
< gmaxwell>
I'll review at least.
< jonasschnelli>
Because both PRs are not downwareds comp., we should bundle them.
< jonasschnelli>
But IMO the chain split is important.
< jonasschnelli>
We shouldn't create to many wallet without the hd change chain
< jonasschnelli>
*wallets
< gmaxwell>
I'm a little concerned that we're thin on user visible features in 0.14, which will make uptake slower, ultimately resulting in slower testing and feedback-- it is what it is.
< jonasschnelli>
Heh. Yes.
< CodeShark>
add some animated gifs :p
< jonasschnelli>
We could change the splash-screen...
< jonasschnelli>
;-)
< jonasschnelli>
Multiwallet would be realtitic if everyone helps reviewing
< gmaxwell>
Replace the logo with a B engraved moon in celebration of the recent price activity. :P
< jonasschnelli>
realistic for 0.14
< jonasschnelli>
bah
< gmaxwell>
I'll see what efforts I can drum up for multiwallet. I agree with that view.
< CodeShark>
thanks for reviving this effort, luke-jr :)
< jcorgan>
i'll throw my vote in for the hd chain split and only deriving keys on the fly
< CodeShark>
I had almost given up hope
< jonasschnelli>
What concerns me with multiwallet is the possible performance downsides.. needs more testing I guess.
< gmaxwell>
jonasschnelli: have you done any work on they keypool management? e.g. removing all the keys up to a wallet observed key to that recovery works better?
< BlueMatt>
gmaxwell: with #9375 alone we could argue "massive improvements in network-propagation speed
< luke-jr>
I'm going to review the HD split stuff when the meeting is over.
< jonasschnelli>
gmaxwell: I did,... but IMO it's non trivial.
< jonasschnelli>
Problem: encrypted wallets
< gmaxwell>
BlueMatt: yes sure, 0.14 is guarenteed to be very attractive to miners already. :) but not much else I think.
< jonasschnelli>
You can't top up the keypool during some internal operations (like SyncTx)
< instagibbs>
alright gotta bounce, tap me for needed review please
< luke-jr>
CodeShark: ☺
< gmaxwell>
jonasschnelli: why is that a problem? You can still remove the keys, even if rescanning isn't magically solved in the first step. (and if the user unlocks before rescan it should be magically solved)
< luke-jr>
jonasschnelli: people can always not-use multiwallet if performance is critical
< gmaxwell>
luke-jr: I think he meant it might have performance regressions on the single wallet case.
< jonasschnelli>
For rescan, yes. But for detecting running with an old wallet (without rescan), its harder
< jonasschnelli>
old wallet = backup
< gmaxwell>
jonasschnelli: oh can't top up during some internal operations... that would be a challenge.
< jonasschnelli>
But agree, we should at least fix the rescan-with-an-old-hd-wallet feature
< jonasschnelli>
I'll work on that.
< jonasschnelli>
But IMO, multiwallet is the lowest hanging fruit for some nice wallet features in 0.14
< gmaxwell>
jonasschnelli: even without fully improving rescan, we should be removing keys from the pool that we know from transactions that they are used. It will help when people have started multiple copies of a wallet concurrently.
< gmaxwell>
okay any other PRs we should discuss briefly for more attention for 0.14 before we move onto proposed topics that don't have PRs yet?
< luke-jr>
gmaxwell: only thing I can think of that could impact it is the lookup of the specific wallet for RPC calls, but I would think that'd be fairly fast
< jonasschnelli>
luke-jr: I think simplest thing would be URL endpoints.
< jonasschnelli>
/<walletA> affects walletA.dat
< gmaxwell>
#action review meeting log and review things people are working on for potentially 0.14.
< CodeShark>
may I talk a little about WitnessMerkleBlocks?
< gmaxwell>
#topic WitnessMerkleBlocks
< gmaxwell>
CodeShark: You have the floor.
< CodeShark>
thank you :)
< CodeShark>
so I started working on a new message type that for filtered blocks that includes the path to coinbase and a partial merkle tree for the witnesses
< CodeShark>
while I don't really like BIP37 at all, we currently lack another query mechanism that doesn't require full block downloads
< CodeShark>
I started working on this code: I started working on a new message type that for filtered blocks that includes the path to coinbase
< CodeShark>
in addition, rather than sending all the transactions as separate messages, it includes them in the same structure
< CodeShark>
while this does not allow for some minor network optimizations, I believe these optimizations are misplaced
< gmaxwell>
I think we like the proof style returns. Does your use case benefit from the bloom style queries? or, for example, would a getblocktxn like query be more useful to you?
< CodeShark>
hmm - good question
< CodeShark>
so getblocktxn would have you specify a block hash and a transaction index?
< gmaxwell>
(I'm not sure you're familar with the BIP152 getblocktxn message, -- it is a "request transactions by their index in a block")
< CodeShark>
oh...I haven't looked at that
< jonasschnelli>
Is this related to the filter commitments?
< gmaxwell>
Yes, you request transactions for a specific block hash by index, in a very efficient way, and get back a single message with the txn packed in it.
< CodeShark>
hmmm - that requires more roundtrips...but it's a more fundamentally important query
< gmaxwell>
So I was thinking before that your particular need for witnesses could be answered by a version of getblocktxn whos reply included the membership proofs. But I wasn't sure if it fully covered your needs.
< BlueMatt>
committed bloom filters could even be deployed w/o soft-forking them in - just have nodes be able to serve them for blocks
< CodeShark>
yes, BlueMatt - I was thinking the same thing
< BlueMatt>
might be expensive to generate, but would be an interesting test
< CodeShark>
I'll have to look at getblocktxn more
< gmaxwell>
In any case, if CodeShark thinks many of the interesting usecases could be answered by a query by index... I would rather work on that first. I am realllly not eager to continue to invest in the pretty much known broken BIP37 approach.
< CodeShark>
I'll definitely take a look
< BlueMatt>
gmaxwell: agreed
< gmaxwell>
(in particular I was thinking you could use BIP37 on the non-witness data, and when you need witnesses, query by index for them)
< CodeShark>
extending BIP37 would be a stopgap measure at best - if we can deploy a better solution nearterm I'm all for it
< gmaxwell>
Sounds good. In any case, I'm glad to help talk it through. I wouldn't want to force you to wait on the BIP37 replacement to do something, but if we can be less stopgap that would be better.
< CodeShark>
excellent :)
< CodeShark>
I'll look into BIP152 a bit more after the meeting
< gmaxwell>
You should still have the option of requesting full blocks... and for privacy reasons I'd really suggest all SPV wallets expose that as at least an option generally. But sure, I agree that we should figure how how to accomidate some kind of sparse fetching for you.
< CodeShark>
I don't really have much more to say on this topic until I have a chance to tinker a bit with that
< gmaxwell>
OKAY, what other topics have I forgotten. We talked some about jonasschnelli's chain split in the prior topic.
< CodeShark>
thanks, gmaxwell :)
< jonasschnelli>
IMO the chain split needs review, thats all. :)
< cfields>
<jl2012> topic suggestion: excessive sighashing protection policy #8755 and #8654. Maybe too late for 0.14?
< cfields>
jl2012: did you have more to discuss there?
< gmaxwell>
Another thing that needs review is #9138 really we don't have enough testing infrastructure for the wallet and fee estimation, so we really do count on eyeballs.
< jl2012>
That's a long post, but it explains the rationale of 8755 and 8654
< luke-jr>
not quite Core-related, but it would be nice to get some BIP Comments posted now that BIP 2 is Active
< jonasschnelli>
I have tagged 8755 and 8564 for 0.14.
< gmaxwell>
jonasschnelli: thanks.
< gmaxwell>
luke-jr: Can you explain briefly what BIP comments are for us?
< jl2012>
8654 is also somewhat a risky refactor
< jl2012>
a bug was found after it got several ACKs
< luke-jr>
BIP Comments are simply put a GitHub wiki page where people can post a brief opinion on the BIP
< gmaxwell>
jl2012: have we at least added a test that would have found that bug?
< gmaxwell>
luke-jr: is there any procedure for providing one?
< luke-jr>
the idea being to provide a central location users can use to differentiate between inadvisable and recommended BIPs
< jl2012>
yes, it now includes the tests to detect that bug
< luke-jr>
gmaxwell: just editing the wiki page, but should ideally be done after the BIP is proposed, and before that point comments ought to be posted to bitcoin-dev instead to allow the drafter to possibly revise the BIP to address concerns
< gmaxwell>
sipa: we've assigned you all the work, so no worries.
< luke-jr>
lol
< sipa>
great
< gmaxwell>
sipa: anything you'd like to discuss?
< jonasschnelli>
morning sipa!
< sipa>
gmaxwell: did you report that the per-txout cache is now a clear performance win?
< gmaxwell>
I thought about that but did not!
< sipa>
so: it turned out that in our earlier benchmark, some debug code was left that resulted in a database read for every txin, which was killing performance
< sipa>
it's now around 30% faster to IBD, with sigchrck disabled, both for small and large fbcache
< sipa>
so... work continues
< sipa>
</report>
< luke-jr>
wow, nice
< gmaxwell>
Further speedups are expected. In one sense this is bad news, since now we'll have to figure out how to make the migration. :)
< luke-jr>
we've required reindexing before
< gmaxwell>
luke-jr: the bigger the chain the larger the pill that is to swallow.
< sipa>
we're investigating better cache eviction policies (instead of the wipe-cache-on-flush method...), no good results yet, but a few ideas remain
< gmaxwell>
esp now that there are pruned nodes.
< luke-jr>
but it's mostly the same problem as IBD for new users
< sipa>
luke-jr: also, pruning is now stable... ideally we eon't require people to redownload to uograde a pruned node
< luke-jr>
true re pruning
< sipa>
anyway, i believe upgrading is theoretically possible, but it's not trivial (chainstate and undo files need processing)
< gmaxwell>
I guess thats an interesting point to bring up: we've found the current wipe behavior is hard to beat, many things we tried that preserved entries were slightly _slower_. It appears that reading is fast and the caches improvement comes largely from preventing writes.
< luke-jr>
upgrading is also consensus-critical, so it will be important to test it well
< sipa>
luke-jr: agree, which is the annoying part
< gmaxwell>
Okay. nothing else?
< luke-jr>
5 minutes left.
< gmaxwell>
okay. I think we can end early. Happy holidays everyone! and if you're travling, travel safely.
< luke-jr>
it will matter when the log is replayed to the bot? :P
< gmaxwell>
#endmeeting
< roasbeef>
as is now, one wouldn't be able to use getblocktxn to fetch the transactions from blocks further than 10 blocks deep afaict
< roasbeef>
will that policy be lifted?
< jonasschnelli>
thanks gmaxwell for hosting.
< roasbeef>
in lightning we'd have another use for it as the outpoints for chanenls are currently communicated using 8bytes the details the blockheight+txindex+txout
< phantomcircuit>
roasbeef, why?
< gmaxwell>
roasbeef: we're not even talking about getblocktxn itself there-- codeshark needs something that returns membership proofs and not just entries.
< gmaxwell>
If there is a utility to making getblocktxn go further back, I think that could be discussed.
< luke-jr>
oh, are we meeting next week?
< luke-jr>
it's not a particularly relevant day in of itself, but it's during Christmas, and many people are probably still doing stuff?
< gmaxwell>
In _general_ we've tried to be conservative about p2p methods that give remote hosts random access to the blockchain; they tend to be DOS vectors (random IO on a 100GB working set) and encourage abuse of the system ("blockchain FUSE module").
< gmaxwell>
Wladimir won't be here for a meeting next week, but I believe I'm willing and able. A lot of people will be out.
< CodeShark>
I've done my traveling for the year already - I'm enjoying some time at home now, so I'll be around :)
< luke-jr>
I can probably be here if we have a meeting, but if a lot of people will be out it may make more sense to just cancel
< gmaxwell>
Okay, lets cancel it then. It isn't like were aren't talking here all the time in any case.
< sipa>
i'll be at 33c3 for next meeting
< gmaxwell>
roasbeef: so you should consider the current limitation on getblocktxn mostly a product of "existing uses had no need for more, and in principle we want to be conservative with this kind of query." rather than any specific need to not permit it.
< waxwing>
sipa: are you giving a talk?
< sipa>
waxwing: no
< waxwing>
sipa: i think they have a Lightning round or something .. insert pun here :)
< bitcoin-git>
[bitcoin] jonasschnelli opened pull request #9408: Allow shutdown during LoadMempool, dump only when necessary (master...2016/12/memp_dump) https://github.com/bitcoin/bitcoin/pull/9408
< gmaxwell>
jonasschnelli: so why can't we repopulate the keypool during SyncTx ?
< jonasschnelli>
it would require an unlock
< jonasschnelli>
(in case of encrypted wallets)
< jonasschnelli>
non-encrypted wallets, fine.
< gmaxwell>
... but what if it is already unlocked?
< jonasschnelli>
Then we could.
< jonasschnelli>
We should do this,... but we can't leave the locked wallets in the dark.
< gmaxwell>
we should do that, then making the rescan work is simply a matter of "unlock first, then rescan"-- not ideal but at least there is a procedure.
< gmaxwell>
Once that works the way to handle rescan is to check if an unlock would be needed and pause the rescan until unlocked. (and in the GUI pop up the password dialog at that point)
< jonasschnelli>
The problem is that there is no interruption point during SyncTx and currently no way to give user a feedback on that level
< jonasschnelli>
Yes. Agree, something like that would be great
< jonasschnelli>
Also, we need to define the gap limit.
< gmaxwell>
Yes, but for rescan we can prompt to unlock before the keypool is empty.
< jonasschnelli>
Which I rather select very high.
< gmaxwell>
well thats just the keypool size. Which I think we should increase to at least 1000, I just haven't pressed for that because we haven't done the chainsplit and privkey on demand changes.
< jonasschnelli>
You don't want to rescan down to genesis
< jonasschnelli>
HD was introduces in 0.13
< jonasschnelli>
Well,.. at least you should have an option to not go down to genesis
< jonasschnelli>
Also, -rescan has no user feedback options, RPC rescan would allow that
< gmaxwell>
I somewhat think that -rescan should be removed in favor of rpc / UI rescan.
< jonasschnelli>
Yes. But most of the others where against that. Well,... pre HD though.
< gmaxwell>
oh? okay. I missed or blissfully forgot that discussion.
< gmaxwell>
We have too many parameters IMO and should constantly look for ones to remove.
< jonasschnelli>
Yes.
< gmaxwell>
rescan rpc could check if the wallet would need to be unlocked (is a locked HD wallet) and return an error, perhaps even have an override if you want to rescan without unlocking (e.g. if your keypool is already big enough)
< jonasschnelli>
Yes.
< jonasschnelli>
Also, we need to be carefull about the lock timeout. :)
< gmaxwell>
I was thinking that perhaps the rescan should just block the timeout.
< gmaxwell>
Though all this talk of rescans makes me feel dirty. Rescans are already a hack, (largely) incompatible with pruning, and unacceptably slow.
< jonasschnelli>
Yes. Just careful to not leave the wallet unlocked in case of en error/execption
< jonasschnelli>
gmaxwell: yes. UTXO rescan should be an alternative
< jonasschnelli>
If you have pruned, you maybe just want to kick out your funds to a new addr.
< jonasschnelli>
(and don't care about tx hist)
< gmaxwell>
yes, though it isn't a good alternative either, because it leaves the wallet in a weird state where it doesn't know the history and doesn't know it doesn't know the history.
< jonasschnelli>
Thats true. But IMO the only option if you prune.
< jonasschnelli>
Execpt your willing to re-download
< gmaxwell>
meaning you might do that with a wallet, then six months later forget that you did that, and then make errors that get you in trouble as a result.
< gmaxwell>
jonasschnelli: sure but we could do better with the UI/RPC to make it clear that the wallet doesn't know the history before a particular point.
< jonasschnelli>
Yes. That's possible.
< gmaxwell>
We can define a "pruned wallet" this is a wallet that only knows transactions up to height X, and knows all its coins.
< jonasschnelli>
Add something in getwalletinfo, ... add something to the UI
< jonasschnelli>
Yes. We should do that.
< gmaxwell>
And then when you want to use the utxo scanning you have to convert your wallet to a pruned wallet. And the UI would show at the earliest position in the tx list ---- history before block X not shown due to wallet pruning ----
< jonasschnelli>
Not sure if we get this into 0.14. Xmas, then feature freeze is close.
< gmaxwell>
and likewise rpc could show a dummy transaction in position 0 to indicate the same thing.
< jonasschnelli>
Yes. I like that.
< jonasschnelli>
Yes. The dummy transaction might work. But careful to not break API consumers.
< jonasschnelli>
Otherwise just getwalletinfo
< gmaxwell>
And anyone could prune an existing wallet... which would probably make a lot of commercial users happy when they've ended up with 500 MB transactions.
< jonasschnelli>
heh. Yes. Ask jouke_ . :)
< gmaxwell>
yea, you're right. better to not do the dummy... just an info field.
< gmaxwell>
er 500 MB wallets.
< gmaxwell>
in any case, this would give a rational way to handle these partial imports that wouldn't result in any surprises.
< gmaxwell>
and "unpruning" a wallet would require a rescan.
< jonasschnelli>
Right. You could switch back anytime.
< gmaxwell>
I dunno if we could do this for 0.14-- some things might be somewhat hard to handle completely, e.g. conflict tracking.
< jonasschnelli>
We could use the auxiliary block requests here
< jonasschnelli>
In case we don't want the re-validate the historical stuff before pruning basepoint. Though, meh.
< gmaxwell>
I don't think we care about revalidating, we already validated it once.
< morcos>
as long as we have these nodes that send headers very fast and then delay sending us the block
< morcos>
9375 is insufficient. i've been trying to test with it and find myself stymied by that problem.
< morcos>
i think even more important than parallel ProcessMessages is the ability to intelligently have more than one block request (of some form) out at the same time
< morcos>
I've been talkign about it with instagibbs, and have in my mind an outline of an idea.. but not 100% clear how to cleanly implement
< morcos>
i think we'll want to be able to have more than 1 (probably 2 is sufficient) PartiallyDownloaded blocks in progress at the same time
< phantomcircuit>
morcos, i dont think you want to have requests out for full blocks in parallel
< phantomcircuit>
just sketches
< morcos>
phantomcircuit: agreed, in short my idea is only if you have no outstanding request will you request a full block
< morcos>
regardless of that, until you have 2 blocktxn requests outstanding, you'll request cmpctblocks from headers or blocktxns from cmpctblocks
< morcos>
or roughly something like that...
< morcos>
sdaftuar also ping on above when you get a chance ^
< BlueMatt>
morcos: yes, initially I had thought that I should avoid PR'ing that until we had parallel process messages for that reason
< BlueMatt>
morcos: however, the caching that that PR does will, by itself, fix a ton of the issues with being slow to send the block
< BlueMatt>
indeed, not entirely, but somewhat
< BlueMatt>
having parallel processmessages should fix much of the test
< BlueMatt>
s/test/rest/
< gmaxwell>
why is reading a block from disk so slow?
< gmaxwell>
no doubt thats part of the reason that rescan is insanely slow.
< BlueMatt>
IIRC we check the merkle tree on it
< gmaxwell>
can we like.. not do that sometimes?
< BlueMatt>
we may no longer
< BlueMatt>
we used to...like 2 years ago
< BlueMatt>
no, nvm, just CheckProofOfWork now
< BlueMatt>
anyway, its slow because deserialize, I'd bet, but havent benchmarked deserialize + build compact block/blocktxn response
< gmaxwell>
well I wouldn't be surprised if it was the deseralize, we know that rescan is quite slow.. but I don't know why.
< BlueMatt>
there's a bench_bitcoin thing for deserialize and deserialize + CheckBlock
< BlueMatt>
its like 10ms
< BlueMatt>
or so
< BlueMatt>
so not incredibly slow, but also definitely not fast
< sipa>
i assume the deserialization is the slowest part
< BlueMatt>
I believe that was the case when i checked
< gmaxwell>
well 10ms * 100 peers is much of the slow results you were reporting. I think you were saying lightsword was seeing 2 second service times?