< jtimon>
between 10775775 and d59a5184 my computer says, but I've risen false alarms before
< jtimon>
but if people want to try to reproduce...again it's walletbackup that fails, with "python3 ./qa/pull-tester/rpc-tests.py" (with or without -parallel=N), but not python3 "./qa/pull-tester/rpc-tests.py backupwallet"
< jtimon>
this happens in d59a5184 but not in 10775775 (to me)
< cfields>
gmaxwell: sounds about right. Excluding the obvious possible serialization pitfalls
< BlueMatt>
cfields: wanna respond to him there? :p
< sipa>
gmaxwell: either he uses default_witness_commitmnt, and he does not need to care about txid or hash or anything
< BlueMatt>
cfields: have anything that needs review? happy to trade for an ack on #9075 :p
< gribble>
https://github.com/bitcoin/bitcoin/issues/9075 | Decouple peer-processing-logic from block-connection-logic (#3) by TheBlueMatt · Pull Request #9075 · bitcoin/bitcoin · GitHub
< sipa>
or he does care about it, and then he needs to compute the witness root himself
< cfields>
BlueMatt: heh, i'd have to find/dust off my bitcointalk credentials
< cfields>
BlueMatt: not yet, I'm rethinking some of mine to take encryption into account. Happy to finish up with 9075 and ack though, I think you already addressed my concerns
< BlueMatt>
cool, I'll move on and look at #8895, then
<@wumpus>
I've had harddisks die on me before, but never in this way, there's a huge range where reading just fails. Usually they just stop spinning up at all.
< sipa>
i don't think my SSD ever spinned at all :(
<@wumpus>
the user needs to provide the momentum
< sipa>
:D
< sipa>
that's very SMART
< paveljanik>
and more fitness :-)
< bitcoin-git>
[bitcoin] paveljanik opened pull request #9121: Initialize variable to prevent compiler warning (master...20161110_rpcdump_uninitialized) https://github.com/bitcoin/bitcoin/pull/9121
< bitcoin-git>
[bitcoin] visvirial opened pull request #9122: fix getnettotals RPC description about timemillis. (master...fix-getnettotals-rpc-desc) https://github.com/bitcoin/bitcoin/pull/9122
< jonasschnelli>
would it make sense to use a 1byte command code instead of the 12byte char in the new bip151 message structure?
< jonasschnelli>
Or would that lead to problems defining new commands and possible overlaps?
< jonasschnelli>
0x01 = inv, 0x02 = getaddr, etc.
< jonasschnelli>
It would just save another 3-11 bytes per message.
<@wumpus>
jonasschnelli: I disagree with doing that: using 12-byte identifiers gives an enormouse scope for different parties to define commands that don't overlap
<@wumpus>
jonasschnelli: using a 1 or 4 byte command code implies a central authority dealing out command codes, and we've had many collisions already
<@wumpus>
(with other things like inv enums)
<@wumpus>
let's not throw away the baby with the bath water while (over) optimizing
< jonasschnelli>
Okay.
<@wumpus>
you could bring it down from 12 to, say, 8 I guess, 64-bit identifiers can be pretty unique
<@wumpus>
heck even 4 bytes probably won't cause a lot of fighting, this is not about about bits but enums
<@wumpus>
but 256 options is just too few and will guarantee fighting, that will end in fire :)
< jonasschnelli>
It looks like that 256 options should be sufficient from the current perspective... but right,.. maybe we better keep it
< jonasschnelli>
An INV would go down from 60bytes to 58bytes. :(
< jonasschnelli>
I guess it's not worth it.
<@wumpus>
INVs can already combine right?
<@wumpus>
ideally send a whole bunch of them at once, not one per message
< jonasschnelli>
A right. INV is a vector.
<@wumpus>
and yes, 256 bytes is 'sufficient from the current perspective' but that's not my point, my point is to make it feature proof and allow permissionless innovation
<@wumpus>
future proof*
< jonasschnelli>
Yes. I think this is wise.
< jonasschnelli>
I wonder if reducing the poly1305 mac tag (currently 16 bytes) would be something that could be considered.
<@wumpus>
is there ever a use case where lots of small messages are sent?
< jonasschnelli>
Probably not.
< jonasschnelli>
wumpus: spv?
<@wumpus>
any that are not vector-able?
<@wumpus>
if so it may make sense to have a 'group' message which just concatenates a bunch of messages (of the same type, say, or at least with types RLE'd), under one mac tag and outer header and such
< jonasschnelli>
Probably not.
<@wumpus>
that would make all messages vector-able. Although it's a design choice at which level to do that...
< jonasschnelli>
Which is different when we look at the implementation
<@wumpus>
oh that's already possible
<@wumpus>
I don't see the problem then, or motivation for looking at few-byte wins in the header
<@wumpus>
the implementation could be smart about it: queue up messages until a certain threshold is reached or timeout
<@wumpus>
like TCP nagle algorithm
< jonasschnelli>
I think switching to encrypted traffic gives us kind of a one-time chance to improve the message structure. I just try to consider every possible optimization.
< jonasschnelli>
Yes. I have discussed that yesterday with cfields. A timeout until messaged gets combined..
<@wumpus>
do beware scope creep
< jonasschnelli>
Good point.
<@wumpus>
in any case: those implementations are possible, they have been done many times before. I wouldn't worry too much about optimizing current implementtions when desiging a protocol
<@wumpus>
optimizing for
<@wumpus>
this needs to be future proof because we don't want to be in the same situation again in say, 5 years
<@wumpus>
also means security should be adequate: don't reduce the MAC tags to near current security bounds
< sipa>
jonasschnelli: openssl just reported a dos attack bug in their chacha20-poly1305, we may want to verify if we'd be susceptible
< sipa>
jonasschnelli: if you want to shave off some bytes, make the command codes variable length
< sipa>
or they could be made lowercase only, and use bitpacking to map 12-character strings to 64-bit values ;)
< cfields>
BlueMatt: ping. I still need help understanding the race comment in #9075.
< gribble>
https://github.com/bitcoin/bitcoin/issues/9075 | Decouple peer-processing-logic from block-connection-logic (#3) by TheBlueMatt · Pull Request #9075 · bitcoin/bitcoin · GitHub
< gmaxwell>
The TCP/IP overhead is already pretty large, so reducing the command size is less relative improvement than you might think.
< gmaxwell>
I think in gneral we'd be better off defining batch operations, like how an inv contains many items.
< gmaxwell>
sipa: lol base-32, 5*12 = 60.
< sipa>
gmaxwell: base 26 * 12 = 57 bits
< gmaxwell>
sipa: you think people complain about endianness, make them pack a non-power of two range... :P
< BlueMatt>
cfields: I think its dependant on multithreaded-peer-logic
< cfields>
BlueMatt: ok, so the issue is that it drops the lock and then re-takes it, and in the meantime, the nodeid could've changed (if multi-threaded) ?
< BlueMatt>
cfields: if you have two block sources at once
< cfields>
right
< BlueMatt>
or, eg, you have a node which gives you a block and then you submitblock the same block
< cfields>
BlueMatt: in that case though, if it's a full block that's submitted, the ban bool could be switched as well, so it'd be possible to skip punishing a peer who's sent a bad full block?
< cfields>
or am i misreading?
< cfields>
(i'm just wondering if it makes sense to switch to a multimap, so that every node gets the correct response)
< BlueMatt>
cfields: huh? no, I'm saying in thread a) a peer adds itself to mapBlockSource as the person to punish, then in thread b) another peer adds itself, then the block is processed from peer a...peer b doesnt get processed as "already rejected" but doesnt get the punishment for invalid block, either
< BlueMatt>
i highly dont think it matters
< sipa>
turn mapBlockSource into a multimap?
< cfields>
sipa: heh, see 3 lines up :)
< cfields>
i just don't see why we'd complicate possible future logic if we could easily guarantee that everyone gets the correct response
< BlueMatt>
huh? I mean if they're on an invalid chain we'll ban them on the next block anyway
< cfields>
BlueMatt: Is there some advantage to leaving it this way? "we'll get it eventually" seems like a strange argument here.
< BlueMatt>
code diff? ease of change?
< jonasschnelli>
sipa: BIP151 has varlen command codes
< sipa>
jonasschnelli: oh!
< sipa>
jonasschnelli: then who cares
< jonasschnelli>
Yes. I think this should be sufficient.
< BlueMatt>
reasons why there is no way in hell we could multithread ProcessMessages in 0.14
< BlueMatt>
:)
< achow101>
oh, meeting already?
< jonasschnelli>
Not sure if it's OT or not, but if possible, it like to propose a short topic "concept of hybrid SPV"
< BlueMatt>
anyone else just want a bloom filter commitment soft fork for that?
< cfields>
jonasschnelli: yes please
<@wumpus>
#topic hybrid SPV
< jonasschnelli>
I wanted to ask if we'd like to see some form of "SPV" (wrong term, i agree) mode in Bitcoin-Core, if it's worth to continue the work
< jonasschnelli>
IMO it could affect the userbase
< kanzure>
was this about initial block download?
< jonasschnelli>
(from expertish to novicish)
<@wumpus>
I think the idea has always been to have some kind of SPV mode in bitcoin core, yes
< sipa>
why would it be a wrong term?
< kanzure>
"SPV" was the one about fraud proofs etc
< sipa>
ok, "client mode" as bitcoin's creator called it then
< jonasschnelli>
I like SPV, ... but some people told me SPV implies bloom filters. I somehow disagree with that though
<@wumpus>
even if full block SPV without bloom filters
< jonasschnelli>
I think full block hybrid SPV sounds ideal IMO
< BlueMatt>
^ that
<@wumpus>
no, it doesn't need bloom filters. Say you want to do SPV against a local node which stores the block chain anyway, there's no need for bloom filters
< jonasschnelli>
Once we have 150/151, we could add bloomfiltering options against bip150 authed peers.
< BlueMatt>
preferably not
< sdaftuar>
jonasschnelli: hybrid spv means it starts out as spv, but eventually becomes fully validating?
< jonasschnelli>
sdaftuar: right.
<@wumpus>
and yes could always be added later, if there is a sane way of filtering that doesn't throw all your privacy out of thew indow
< BlueMatt>
better designs are possible, but for full blocks agreed
< sdaftuar>
jonasschnelli: sounds awesome!
< NicolasDorier>
I'm bit lost here, why is it called SPV if the node store the blockchain oO
< BlueMatt>
NicolasDorier: SPV here is referring to the trust model of trusting miners
< NicolasDorier>
is there some past discussion somewhere about it I can read ?
<@wumpus>
NicolasDorier: no, it doesn't need to store it
< kanzure>
the level of confusion around naming and name-implications is really high. it's sort of amusing.
<@wumpus>
NicolasDorier: it just receives the blocks, filters them locally intead of setting a bloom filter
< jonasschnelli>
Haven't got conceptual ACKs and wasen't sure if its worth to continue, ... make it clean and stable, etc.
< NicolasDorier>
how is it different from a pruned node?
< kanzure>
if it's not immediately obvious to most of us that feature x is included (like block downloading or something) then we need better names :P
< jonasschnelli>
NicolasDorier: it does no validation
<@wumpus>
NicolasDorier: it would allow receiving transactions while validation is not complete yet, for example
< jonasschnelli>
NicolasDorier: just header chainwork check and pass the transaction to the wallet
<@wumpus>
a pruned node is a full node
< NicolasDorier>
ah ok, so basically a node without UTXO ?
< jonasschnelli>
The main difference is, that we load blocks during IBD first from the wallet brthday.
< jonasschnelli>
NicolasDorier: yes. No UTXO set
< jonasschnelli>
No mempool, same fee problem that all SPV wallets have to deal with
< NicolasDorier>
ah ok got it
<@wumpus>
pruning has nothing to do with SPV, full node has nothing to do with 'storing the block chain' or not
< jonasschnelli>
But with the option of slowing becomming a full node
< btcdrak>
jonasschnelli: I dont think anyone concept ack because it's so obviously a good feature.
<@wumpus>
right, it is about the UTXO set
<@wumpus>
jonasschnelli: I've been very busy, sorry
< jtimon>
oh, I undesrtand the details now, headers first, then you are an spv node that fetches the full block instead of using bloom filter but you keep syncing on the background to become a full node, sounds all fine
<@wumpus>
jonasschnelli: it's obviously a good idea
< BlueMatt>
jonasschnelli: but still fill-blocks-in-background?
< BlueMatt>
or are we moving away from that?
< BlueMatt>
but, yes, obviously good
< jonasschnelli>
BlueMatt: there are two modes, purespv and hybridspv
<@wumpus>
BlueMatt: that should certainly be an option
< jonasschnelli>
purespv will just throw away blocks...
< BlueMatt>
oh, only optional? hum, not a fan of it only being optional
< jonasschnelli>
hybrid spv will keep the blocks and does IBD in the background (maybe throttled)
<@wumpus>
hybrid stores them for later
<@wumpus>
BlueMatt: why not?
< morcos>
jonasschnelli: +1 on the ability to throttle, think that would make it very useful
< BlueMatt>
i mean you dont have to keep blocks, but you still want to ibd and build the utxo set
< BlueMatt>
/validate
< BlueMatt>
wumpus: because if you're not gonna be upgrading to full trust model why are you running bitcoin core as a wallet?
<@wumpus>
being able to use bitcoin core as a full SPV node would be useful, especially with a local node that does validate
<@wumpus>
BlueMatt: why not?
< jonasschnelli>
BlueMatt: if you throw away blocks with the long term goal of getting a full node your wasting bandwith
< BlueMatt>
jonasschnelli: no, you're not!
< BlueMatt>
jonasschnelli: you're upgrading your security model
< jonasschnelli>
BlueMatt:: you have to download the block again!
< BlueMatt>
jonasschnelli: huh? oh, you mean keeping the blocks at the tip that you're using to fill wallet...ahh, misunderstanding
< jonasschnelli>
purespv is interesting when connecting to a trusted node.
<@wumpus>
right
< BlueMatt>
I'm not sure if its worth the effort to make bitcoin core a competitive spv wallet - they mostly all already support connecting to a trusted node (though need auth)
< jonasschnelli>
fullblock SPV gets uninteresting if you import keys or watchonlys.. because we set the timestamp to 0 = download the whole blockchain
< achow101>
so hybrid keeps blocks for later and purespv doesn't? Is that the only diference?
< BlueMatt>
if someone wants to work on it I wont complain, but...
< jonasschnelli>
achow101: right.
< BlueMatt>
jonasschnelli: argh, I'm lost in your terms here
< btcdrak>
Yeah I also think purespv seems wrong for Bitcoin Core.
< btcdrak>
but w/e
< sipa>
i don't see why
< jonasschnelli>
achow101: hybrid means, your doing IBD with all its downsides...
< NicolasDorier>
well bitcoin core has a wallet part
< sipa>
bitcoin core has a perfectly good wallet
< BlueMatt>
purespv? fullblock spv? define?
< sipa>
seems stupid to restrict it to full node use only
< CodeShark>
We really should drop the term spv
< jtimon>
achow101: it seems also hybridspv does background ibd while pure doesn't
< achow101>
ok
< jonasschnelli>
purespv = no IBD, hybridspv = SPV during IDB
< BlueMatt>
jonasschnelli: wait, so what is fullblockspv?
<@wumpus>
both
< btcdrak>
I thought the point of the hybride SPV is just to make the wallet usable immediately during IDB, and put an end to the poor user experience for new users.
< BlueMatt>
huh
< jonasschnelli>
Both. Yes.
< BlueMatt>
what does both mean?
< jtimon>
isn't hybridspv the same as "both"?
< sipa>
BlueMatt: both hybrid and pure download full blocks
<@wumpus>
no bloom filters are used in either case, it is all retrieving full blocks
< jonasschnelli>
Both modes (pure and hybrid) are full block SPV modes.
< achow101>
fullblock spv is where the full block is downloaded and the transaction is pulled. watever happens to the block is depended on hybrid or pure
<@wumpus>
what is so hard to understand about that?
<@wumpus>
achow101: right
< jonasschnelli>
let me write a short post on bitcoincore ML
< BlueMatt>
wumpus/sipa: ahh, just referring to the way we download in either case
< BlueMatt>
wumpus: the way it was referenced it sounded like a different mode
< jonasschnelli>
From the enduser perspective, the modes are very different
<@wumpus>
the only difference is that in pure SPV mode, the blocks are thrown away and there is no IBD, in hybrid mode the blocks are backfilled and IBD happens
< BlueMatt>
wumpus: yes, i was confused and thought there was a third mode
< jtimon>
I see no problem with allowing purespv optionally, the default is going to be hybridspv, right?
<@wumpus>
ok
<@wumpus>
jtimon: yes
< ryanofsky>
in the pr, i suggested calling hybrid spv "prioritized download"
< jonasschnelli>
purespv can solve an interesting usecase where one likes to decouple the wallet from the node
<@wumpus>
jonasschnelli: exactly
< BlueMatt>
jtimon: agreed, just saying that it seems a strange thing to focus work on...there are already good options for consumers on the purespv front
< BlueMatt>
like, lots of good options
<@wumpus>
jonasschnelli: it's basically a stand-alone wallet
< jonasschnelli>
BlueMatt: purespv would be the only fullblock SPV wallet in the wild
< jtimon>
BlueMatt: I suspect the reasoning is that it will be relatively cheap to add that extra mode
<@wumpus>
BlueMatt: it is not *focusing* work on, it's just a by-effect of allowing it
< BlueMatt>
wumpus: yup, I'm fine with it being a free feature, just making a note
< jtimon>
maybe a pr with the hybrid default one and a later one adding the purespv mode would make sense
< achow101>
purespv would be great for privacy
< jonasschnelli>
not for bandwidth thought
< sipa>
BlueMatt: fair enough, but if we want hybrid spv because of the terrible syncing experience, we have all the pieces in place for purespv as well
< CodeShark>
I still don't quite get purespv - sorry, scrolling back and keeping up is hard
<@wumpus>
sigh.\
< jonasschnelli>
purespv = plain full block SPV
<@wumpus>
any other topics?
<@wumpus>
we're starting to repeat ourselves and that is annoying
< gmaxwell>
hah
< jonasschnelli>
download block, no validation, extract relevavnt txs
< CodeShark>
ok
< jonasschnelli>
Okay. I'll continue and try to make small PRs (could be hard though). Thanks
<@wumpus>
please just read back if you want to know things that have been discussed before, we can't repeat every single thing specifically for everyone
< BlueMatt>
topics?
< MarcoF____>
topic suggestion: 0.14
< CodeShark>
I'd rather discuss bloom filter commitments or clientside bloom filtering from trusted nodes
<@wumpus>
#topic multithread ProcessMessages
< MarcoF____>
Personally I mostly care about multiwallet refactoring for 0.14
< MarcoF____>
but there are plenty of other pulls open, so we might want to prioritize?
<@wumpus>
BlueMatt ^^
< BlueMatt>
i mean i think we all want multiple message-processing threads
< jonasschnelli>
+1
< gmaxwell>
Sorry, I delayed matt by asking him about it offline. My only comment was "don't reorder messages from a single peer!" :P but apparently I'm behind the times.
< BlueMatt>
and while it probably wont be so useful with cs_main at the head of like every message, I'd like to see plumbing for it sooner rather than later
<@wumpus>
well if cs_main usage is brought down enough, that will start to make sense
< BlueMatt>
then people can remove cs_main one message at a time and it will be useful
< BlueMatt>
I'd be happy to see, but dont know if we'll make, cs_main split-outs too much for 0.14 (thats next after main.cpp split)
< BlueMatt>
but I'd like to see a few free wins like being able to respond to getblocktxn requests while a block is processing
< gmaxwell>
okay, thanks for the concrete example of something pretty useful.
< BlueMatt>
anyone have any thoughts on blockers for this? things likely to break? etc?
< cfields>
BlueMatt: seems unlikely to have any real-world benefit without breaking out CNodeState? Maybe we should try to knock that out first?
< gmaxwell>
I was struggling to come up with one, but that one is good.
<@wumpus>
being able to service multiple nodes at once would be very useful
< BlueMatt>
cfields: why do you have to break out CNodeState? you'll never call ProcessMessages in two threads for a single peer at the same time
< BlueMatt>
cfields: that would violate our serialization stuff
< gmaxwell>
Yes, in particular being able to reply to getblocktxn multiple times in parallel should reduce block relaying delays.
< BlueMatt>
gmaxwell: yes, i really want it for FIBRE-based relay network - respond to getblocktxn from a shared_ptr<const CBlock> block_currently_processing
<@wumpus>
I mean looking at it from the other side, there's no good reason for keeping doing message processing only in one thread
< morcos>
BlueMatt: I think its going to be important to do a thorough review of synchronization issues first. Sometimes I feel like things just happen to not be a problem b/c they are only accessed from the single thread.
< BlueMatt>
morcos: yea, thats my concern :(
< cfields>
BlueMatt: well when each one is grabbing for cs_main, it only takes one long lock to drop us back to single-threaded
< morcos>
We do an ok job of fixing these missing locks when we find them, but if we're goign to make multiple message processing threads, we need to make sure we've really got them all
< BlueMatt>
morcos: any concrete things i should be looking for
< morcos>
I think we should make a list of what needs to be protected by cs_main
< BlueMatt>
cfields: yea, I mean I'm ok with switching to a multi-threaded model that does ~nothing and then slowly reducing the locks
< gmaxwell>
So I think making process message concurrent may create greater exposure for data races around the nodestats. Right now the locking around stats is _widely_ incorrect, leading to undefined behavior. (I haven't been watching closely and cfields likely improved a lot of it recently, but I expect there are still problems)
< gmaxwell>
(this is responsive to matt's question about likely sources of problems)
< sipa>
gmaxwell: for stats we can just change them all to atomics
< gmaxwell>
I recommend that we run testing harnesses with valgrind DRD and try to get this change to be data race free at least according to DRD.
< BlueMatt>
gmaxwell: good suggestion, indeed
< gmaxwell>
sipa: I made that suggestion previously. I think we access them infrequently enough that it's not an awful idea.
< sipa>
gmaxwell: you can also cache per peer, and flush to a locked global occasionally if it's a concern.
< sipa>
stats are easy, no consistency requirements
< gmaxwell>
last time I ran DRD I saw some races around stats but there wasn't a wall of errors.
<@wumpus>
and the per-node stats should not be an issue as we likely don't want to be processing two messages from the same peer at the same time?
< cfields>
sipa: that sounds like a good model
< sipa>
wumpus: indeed
< BlueMatt>
wumpus: yes, that would reduce lots of issues
< gmaxwell>
wumpus: I suspect we can get a message from peer A that causes use to iterate over all the other peers stats.
< sdaftuar>
BlueMatt: have you given much thought to how the threading logic would work?
< sdaftuar>
i mean, how you'd decide how to split work across threads
< gmaxwell>
(just an example why single peer at a time doesn't automatically mean thread safty of per node stats)
<@wumpus>
gmaxwell: that sounds scary
< gmaxwell>
(maybe we don't actually do that anywhere, at least in response to a message... a connection, for example, causes that--)
< BlueMatt>
sdaftuar: hum? do we need anything more complicated than just a thread pool each looking for peers that have available work and arent being worked on that just wait on a cv when there is nothing to do
<@wumpus>
gmaxwell: no, but keeping to rules like that will make it easier manageable
< gmaxwell>
Another point to keep in mind is thread prolifervation and address space usage on 32-bit hosts. Not a reason to not do this, but just a cost.
< sdaftuar>
well, my thought was that since most messages are tx's, we might find ourselves tying up all our threads but unable to continue, as they're all contending on cs_main
< sdaftuar>
so you wouldn't necessarily be able to respond to a getblocktxn while processing a block
< sdaftuar>
you could do somethign smarter though...
<@wumpus>
gmaxwell: it would still be possible to run with only one thread I suppose
<@wumpus>
gmaxwell: I'm not terribly worried
< BlueMatt>
sdaftuar: yes, i was thinking for v1 we ignore that issue
< sdaftuar>
ok :)
< gmaxwell>
wumpus: I'm not either.
< BlueMatt>
sdaftuar: indeed, eventually we could have some "oh, these messages take cs_main" list to make it smart
< gmaxwell>
Besides, this is a better use of threads (actual concurrency) than some others we have had.
< BlueMatt>
gmaxwell: very true :(
<@wumpus>
gmaxwell: we certainly shouldn't be restricting work towards better performance because there happen to be a few nodes on small systems, those can be handled specifically
< gmaxwell>
(e.g. connect has a thread, wallet flush thread, etc.)
< BlueMatt>
we can make up the difference by moving wallet flush into cscheduler thread :)
< gmaxwell>
wumpus: agreed, I am just exposing areas of consideration. I support the work.
<@wumpus>
(and I run most of my nodes on 32-bit ARM so it's not like I don't care)
< BlueMatt>
well, and like three net threads to cscheduler
<@wumpus>
yes, we can do thread-stack accounting like that, but let's not get lost in details
< morcos>
and while we're at it we can increase script checking threads :)
< petertodd>
wumpus: oh, on rpi's and the like?
<@wumpus>
petertodd: cubox-i's are my favourite
< petertodd>
wumpus: interesting, thanks!
<@wumpus>
next topic, I suppose
< gmaxwell>
(there are other possibilties like decresing the stack size on some threads that don't do much, firefox uses 128k or 256k (I forget) stacks for media processing threads. So I really don't at all think it's a blocker, just something to keep in mind.)
<@wumpus>
gmaxwell: indeed you can do that through an environment variable IIRC, I think I even list it in my "reducing bitcoind memory usage" document ,if not I should
<@wumpus>
#topic 0.14
<@wumpus>
@MarcoFalke
< jonasschnelli>
multiwallet support would be nice
<@wumpus>
oh he left?
< gmaxwell>
wumpus: yes, well you can control it for all threads in an env variable --- though that can result in security problems. :( But there are some threads where their stack usage is precisely decidable easily. (e.g. the connect thread).
< jtimon>
yep, it seems so
< jonasschnelli>
[20:36:29] <@wumpus>#topic 0.14
< jonasschnelli>
[20:36:49] <@wumpus>@MarcoFalke
< BlueMatt>
MarcoFalk_:
< bitcoin-git>
[bitcoin] morcos opened pull request #9123: Remove extraneous LogPrint from fee estimation (master...eliminateFeeWarning) https://github.com/bitcoin/bitcoin/pull/9123
< MarcoFalk_>
Jup, I'd like to hear what is a priority to get in to 0.14
< jtimon>
multiwallet refactoring for 0.14
< MarcoFalk_>
^
<@wumpus>
gmaxwell: I'm not sure about security problems, though yes it can cause crashes if you set it that low that the stack underflows
< * jonasschnelli>
gives MarcoFalk a IRC bouncer for birthday
< BlueMatt>
main.cpp split, but thats guaranteed at this point, pretty much
< sdaftuar>
i think the validation speedups from jeremyrubin would be great for 0.14
< MarcoFalk_>
Ok, what is the progess on the net refactor. Is it almost done?
<@wumpus>
gmaxwell: then again the thread stacks are extremely small on some platforms
<@wumpus>
gmaxwell: linux assigns quite a lot by default
< BlueMatt>
sdaftuar: yea, at least the cucku cache thing
< jonasschnelli>
I think there is not much left for the multi-wallet support. Though, not sure if we can make this happen for 0.14
< jonasschnelli>
Also.. we should add a private-key free mode.
< jonasschnelli>
A safe-mode
< gmaxwell>
sdaftuar: I agree, all of them.
<@wumpus>
if we all agree to start reviewing wallet changes, I'm sure we can get multi wallet support in 0.14
< jtimon>
I would like to move on libconsensus, but so far nothing seems to be happening on that front
< MarcoFalk_>
Some of the wallet changes need rebase
< jonasschnelli>
wumpus: Nice! I can review some stuff next week.
< gmaxwell>
In terms of wallet features I think some kind of multiwallet support would have the greatest ratio of benefit to cost+risk.
< gmaxwell>
It's just software eng.
< jonasschnelli>
gmaxwell: ack!
< jonasschnelli>
Also, with the current fundrawtransaction options, you can perfectly run watch-only wallets
< cfields>
I'm aiming to get net.h/cpp split in half in the next ~week also.
< gmaxwell>
So I think multiwallet should be a greater priority. Also while 0.14 has a lot of really great infrastructural changes, I think it has relatively few ordinary user visible improvements.
< paveljanik>
Network on/off in GUI please :-)
< jonasschnelli>
paveljanik: Yes. I'll make that happen soon.
<@wumpus>
because most of the focus has been on 0.13.1, 0.14 will be somewhat of a smaller release, I don't think that's a problem
< gmaxwell>
There are a number of other things I'd like to work on, like more bandwidth usage controls. Improvement to header fetching logic... but I think it's not useful to speak to work that hasn't even started.
< jonasschnelli>
I was hoping the mempool stats can be in 0.14... but not sure if there are enough reviews
< BlueMatt>
bumpfee
< MarcoFalk_>
I was pinging luke-jr on #8776, but haven't heard anything about luke-jr recently
<@wumpus>
we know about them, what they need is more review and testing :)
< MarcoFalk_>
(and rebase) *ducks*
< morcos>
I'm going to do at least some minor (and needed) improvements to fee estimation. Is there any interest on estimates for longer than 25 blocks? Or should I continue to punt on that?
< cfields>
wumpus: yes, i don't think that's likely for 0.14 though. The libevent changes are going to be quite different from what I originally intended, I think
< gmaxwell>
WRT review, I believe that multiwallet review is fundimentally easier than bumpfee for example, because in bumpfee I need to reason about a complex state machine, the effect on the network, corner cases with double spends, etc. And multiwallet is 98% "does this crash or not". :)
<@wumpus>
cfields: it's taking quite longer than we expected :)
<@wumpus>
gmaxwell: so start reviewing multiwallet pulls then :D
< BlueMatt>
wumpus: lol, doesnt everything
<@wumpus>
(or did you already?)
< gmaxwell>
morcos: mempool backlog levels are not great enough where estimates >25 blocks matter that much I think, currently.
< jtimon>
btw maybe we can talk about the "custom mode" (separately from the blocksigning mode which I think it's unlikely to get to master with the union and the global, but we can just rebase every release)
< cfields>
wumpus: yes, sorry for that. I vastly underestimated the refactor impact.
<@wumpus>
BlueMatt: yes, I'm kind of overwhelmed already, so I don't care that much some things are going slower
< morcos>
gmaxwell: But its often possible to pay a much lower fee if you are willing to wait 100 blocks or whatever... So in the interest of keeping fees down.., but I just don't know if thats much of a use case.
< gmaxwell>
morcos: interesting, if it would make a big difference I think it would be interesting. I wasn't thinking it would make much of a difference.
<@wumpus>
cfields: my worry is that some things are blocked on it, e.g. gmaxwell talks about wanting to do things with bandwidth management, those things would all be easier if we had switched to libevent, instead of trying to cram it into the current networking backend
< gmaxwell>
morcos: I still wouldn't priortize it over general improvements.
< morcos>
gmaxwell: tremendous difference. i'm pretty sure you could always pay about 2 sat/byte if you are wiling to wait a couple days.. limiting factor might be the 3 day eviction.
< gmaxwell>
Yes, I've held off on doing too much more with bandwidth management due to that. There are somethings we can do, for example delaying inv relays when bandwidth starved.
< cfields>
wumpus: ACK. I'll pick up the pace.
< gmaxwell>
morcos: my publically reachable nodes don't really expirence 3 day eviction, -- some clown or another inevitably connects and restores the evicted stuff. :P
< sipa>
gmaxwell, cfields: it sounds to me like the bandwidth management you're talking about is very different and probably won't have much code conflicts
< sipa>
as gmaxwell is talking about application level decisions that afaik don't even affect anything at the network layer
< gmaxwell>
Well I specifically just mentioned the part of the management scope that wouldn't have conflicts. (and I think should probably be done first: to quickly offer something but then relay it out slowly is somewhat peer abusive. :P )
< cfields>
sipa: indeed
<@wumpus>
other topics?
< gmaxwell>
we should say hello to all the americans that missed the timezone change.
< gmaxwell>
and are just arriving now. :P
< jcorgan>
heh
< petertodd>
gmaxwell: canadians too :)
< gmaxwell>
Hi guys.
< achow101>
hi
< gmaxwell>
Welcome to the end of the meeting.
<@wumpus>
yes, welcome
< sipa>
kthxbye
<@wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Nov 10 19:55:51 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< bitcoin-git>
[bitcoin] paveljanik opened pull request #9124: Use better name for local variable to prevent -Wshadow compiler warning (master...20161110_Wshadow_benchcheckblock) https://github.com/bitcoin/bitcoin/pull/9124
< sdaftuar>
wumpus: i think it's good to go, but thought bluematt might want to review since it is a (very small) change to BIP 152
< bsm1175321>
What is the plan for the MSG_FILTERED_WITNESS_BLOCK message? (spv + segwit...)
< bsm1175321>
or block type rather in inv/getdata
< gmaxwell>
bsm1175321: for most SPV wallets the witness is useless and not desirable.
< gmaxwell>
Without the pubkey they can't verify it.
< gmaxwell>
bsm1175321: I think many people would be glad to work on a spec for that with clear applications.
< gmaxwell>
But I expect most (though not all) SPV segwit wallets to not use it.
< bsm1175321>
Which pubkey? The witness contains the pubkeys no?
< gmaxwell>
The scritpubkey.
< bsm1175321>
I was surprised to find the bcoin library setting this flag on testnet...spent a day tracking down why 0.13.1 wasn't responding to my getdata...
< gmaxwell>
Setting what flag?
< bsm1175321>
MSG_FILTERED_WITNESS_BLOCK
< bsm1175321>
on getdata queries
< bsm1175321>
Clearly a bug for bcoin...anyhoo
< bsm1175321>
What do you have in mind for "a spec with clear applications"?
< tulip>
bsm1175321: similarly BIP37 hashes items in the scriptpubkey individually and adds them to the filter.
< tulip>
MSG_FILTERED_WITNESS_BLOCK would regrettably have to do the thing where the witness of a matching transaction is serialised and inserted back into the filter as well.
< bsm1175321>
Woah I had no idea the BIP37 filters are applied to *any* data element in your script.
< tulip>
the transaction, the TXID, the serialised inputs, every data element in any script.
< Chris_Stewart_5>
tulip: I don't think the entire txs is checked, and I think it is only the outpoints in the inputs?
< Chris_Stewart_5>
+ date elements in the script like you said
< tulip>
Chris_Stewart_5: you're right that it doesn't match the entire transaction, just the hash. I think the other parts of my comment are correct looking at the specification.
< gmaxwell>
yea, that 'feature' really contributes to the lack of privacy and utiliy of BIP37.
< FreeUser>
Hello.
< FreeUser>
The alert system will be removed, yes?
< FreeUser>
But how users will receive info about critical bugs?
< gmaxwell>
it was removed a long time ago, it is only now being deactivated for older software.
< gmaxwell>
FreeUser: same way they recieve news about anything.
< FreeUser>
How to disable alerts in old Bitcoin Core versions?
< gmaxwell>
FreeUser: there is a final alert which will be sent which causes a final alert to be displayed which cannot be overridden by any other alert.
< FreeUser>
Are alerts stored in blockchain?
< gmaxwell>
No.
< FreeUser>
But how users received these alerts?
< gmaxwell>
bsm1175321: there is only one application that I'm currently aware of for a SPV wallet to see witness data: so a multisignature wallet can track which of the signers signed their own coins.
< gmaxwell>
FreeUser: they were sent peer to peer between indivigual nodes.
< FreeUser>
What will happens with Litecoin/other altcoin?
< FreeUser>
*altcoins
< gmaxwell>
nothing, they're totally seperate systems
< FreeUser>
Why Litecoin not updating Litecoin Core?
< bsm1175321>
gmaxwell I'm interested in SPV clients getting witness data because I'm using the spent pubkeys for out-of-band auth and communication.
< FreeUser>
Is it hard to fork newer version and change it like old?
< sipa>
FreeUser: ask them
< sipa>
litecoin is offtopic here
< gmaxwell>
bsm1175321: the public keys are in the paying transactions txouts.
< FreeUser>
What is difference between 0.13.1 and 0.13.99?
< FreeUser>
And why 99?
< sipa>
FreeUser: 0.13.99 is the version number used in the 0.14 branch before 0.14 is released
< sipa>
FreeUser: at this point we have 0.14 branch with more significant changes, and a stable 0.13 branch that only receives bug fixes
< FreeUser>
Is Bitcoin Knots an official wallet?
< sipa>
there is no 'official'
< sipa>
Knots is a fork of Bitcoin Core, maintained by different people
< FreeUser>
Official means from Bitcoin Core devs
< sipa>
i'm not involved in Knots
< gmaxwell>
whats a Bitcoin Core devs? there is no such thing as membership.
< sipa>
and i develop for Core.
< bsm1175321>
gmaxwell: correct, I'm interested in obtaining that paying transaction, with witness data, by SPV clients.
< gmaxwell>
Perhaps you're a Bitcoin Core dev, have you submitted a patch? :)
< Chris_Stewart_5>
Damn! I paid that guy all my doge coin for a bitcoin core dev sticker..
< FreeUser>
I can see "Bitcoin Core Developers" while loading Bitcoin Core.
< sipa>
FreeUser: you too can contribute to either, none, or both
< gmaxwell>
FreeUser: yes, that just means all the people who have contributed.
< MarcoFalke>
FreeUser: The "Bitcoin Core Developers" is just the short version of all the credits (which you can find in the release notes)
< FreeUser>
Can I find Satoshi Nakamoto in release notes?
< sipa>
he has not contributed to any release since 0.3.19
< sipa>
(at least not under that name)
< Lauda>
I'm somewhat inclinded to believe that sipa is satoshi
< Lauda>
they both start with an 's'.
< FreeUser>
Can Satoshi Nakamoto be killed?
< FreeUser>
So
< sipa>
wth are you talking about?
< Lauda>
Oops wrong chat, thought this was #bitcoin. Sorry.
< sipa>
please take this elsewhere
< FreeUser>
Some cracker cracked his P2P Foundation account.
< sipa>
off topic
< sipa>
you're free to ask questions about Bitcoin and Bitcoin Core's development model here
< * gmaxwell>
waits for the rbtc headline
< Satoshin>
Hello.
< Satoshin>
.help
< gmaxwell>
I recommend setting a wildcard on that.
< BlueMatt>
sdaftuar: sounds good, will put it on the review-stack :)
< BlueMatt>
sdaftuar: up next after cuckoo
< cfields>
hmm, is it intended that feeler connections are allowed to send more than just version messages out?
< gmaxwell>
cfields: not really, though I noticed that to and concluded that we'll probably like to expand their behavior over time.
< cfields>
gmaxwell: ok. I suppose at worst, it just makes it look more like a normal node
< gmaxwell>
E.g.e treat them more like like regular connections, and potentially disconnect another connection that hasn't been so useful.
< cfields>
gmaxwell: right. I kinda assumed they did some of that already. For ex, noting nodes with unexpected services in addrman.
< cfields>
Seems a shame to learn that, then mark it as up/fresh anyway
< gmaxwell>
they do update the services before disconnecting. (though we avoid trying to feeler to things without relevant services right now, which diminishes the usefulness somewhat. :) )
< cfields>
ok, roger. As long as it's not unintentional behavior :)