< bitcoin-git>
[bitcoin] jonasschnelli closed pull request #10517: Factor out CCoinsView based AreInputsStandard/IsWitnessStandard (master...2017/06/policy_compile) https://github.com/bitcoin/bitcoin/pull/10517
< bitcoin-git>
[bitcoin] jonasschnelli closed pull request #10238: Change setKeyPool to hold flexible entries (master...2017/04/keypool_fix_a) https://github.com/bitcoin/bitcoin/pull/10238
< bitcoin-git>
[bitcoin] jonasschnelli closed pull request #10251: Add balances cache / GUI: use a signal instead of a poll thread (master...2017/04/gui_rm_locks) https://github.com/bitcoin/bitcoin/pull/10251
< bitcoin-git>
bitcoin/master 58d91af MeshCollider: Fix race for mapBlockIndex in AppInitMain
< bitcoin-git>
bitcoin/master 35aeabe MeshCollider: Make fReindex atomic to avoid race
< bitcoin-git>
bitcoin/master 731065b MeshCollider: Consistent parameter names in txdb.h
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #11107: Fix races in AppInitMain and others with lock and atomic bools (master...fix_mapBlockIndex_race) https://github.com/bitcoin/bitcoin/pull/11107
< deltaT>
simple question i hope, when i created my bitcoin core wallet i didnt write down the private key, how do i find it now?
< jonasschnelli>
deltaT: which Bitcoin Core version?
< jonasschnelli>
deltaT: did you lost you wallet.dat? file.... also, this question should be asked in #bitcoin (this here is the development channel)
< deltaT>
0.14.2
< BlueMatt>
wumpus: #9572 looks relatively merge-able...it is consensus so if you want to ack first that'd also be nice, but its simple and has 4 already
< bitcoin-git>
[bitcoin] mess110 opened pull request #11455: CTxMemPool::GetMinFee should not return CFeeRate(0) (master...fix_mempool_GetMinFee_bug_returning_below_minRelayTxFee) https://github.com/bitcoin/bitcoin/pull/11455
< bitcoin-git>
[bitcoin] TheBlueMatt opened pull request #11456: Replace relevant services logic with a function suite. (master...2017-09-service-flags-cleanups) https://github.com/bitcoin/bitcoin/pull/11456
< bitcoin-git>
[bitcoin] mess110 closed pull request #11410: [rpc] [tests] mempoolminfee should not drop below minRelayTxFee (master...add_minrelaytxfee_to_getmempoolinfo) https://github.com/bitcoin/bitcoin/pull/11410
< tloriato>
Good afternoon. I was looking into BIP45, that tries to standardize the Structure for Deterministic P2SH Multisignature Wallets, but I've read the emails discussing it and seems to have a little or disagreement between the need for this particular BIP. I'm wondering if there is another standard way to generate a Deterministic Multisig Wallet? I was inclined to generate a standard Mnemonic (BIP39) and split it using Shamir
< BlueMatt>
sipa: did you find time to write up the segwit wallet tradeoffs between the various ways of doing it and making an argument for why your pr is your preferred version?
< BlueMatt>
sipa: I believe you said you were gonna do that last meeting
< jonasschnelli>
tloriato: AFAIK BIP45 is the only "standard" to create multisig addresses with a set of given pubkeys.
< bitcoin-git>
bitcoin/master 4f890ba Donal OConnor: Add new step to clean $PATH var by removing /mnt specific Window's %PATH% paths that cause issues with the make system
< bitcoin-git>
bitcoin/master 696ce46 fanquake: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04
< bitcoin-git>
bitcoin/master becbd71 Wladimir J. van der Laan: Merge #11437: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04...
< bitcoin-git>
[bitcoin] laanwj closed pull request #11437: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04 (master...windows-build-1704) https://github.com/bitcoin/bitcoin/pull/11437
< jl2012>
why couldn't we just have a PrecomputedTransactionData for each transaction?
< sipa>
we do?
< sipa>
that's just the vector that holds them
< jl2012>
yes, but why need to hold them all?
< sipa>
variables need to exist somewhere...
< jl2012>
yes, I mean, why not just have a line PrecomputedTransactionData txdata(tx); inside the loop "for (unsigned int i = 0; i < block.vtx.size(); i++)"?
< sipa>
jl2012: oh, i read "why need to them *at* all", sorry
< sipa>
the reason is that they're used from other threads during parallel validation
< jl2012>
ok...so without the vector, each thread will re-compute the hashes once?
< sipa>
depends how you do it
< sipa>
if you do what you suggest, you'd get undefined behaviour
< jl2012>
why?
< sipa>
as the precomputation data would be out of scope before it's accessed from the validation threads
< sipa>
most of the validation happens after that loop exits, and before the queue's wait call returns
< JeremyRubin>
jl2012: I have a patch that makes them shared_ptrs
< BlueMatt>
sipa: not anymore (tx script caching) :p
< JeremyRubin>
conceptually, that's what you want
< JeremyRubin>
performance wise, a vector is better
< jonasschnelli>
gmaxwell: for the notification system, you had concerns about the reliability.. would a long poll queue that only removes elements from the queue on an ack from the client be something that would defeat your concerns?
< jonasschnelli>
With that concept, loosing notifications would be very unlikely
< sipa>
BlueMatt: ok, i reformulate - most of the validation, if it happens, happens after that loop exits
< jl2012>
oh, I thought all script validation is done with CheckInputs?
< sipa>
jl2012: yes and no
< sipa>
CheckInputs returns a vector of CScriptCheck objects, which represent validation that will happen on another thread
< sipa>
those get pushed to the checkqueue, where validation threads pick them up
< jl2012>
ontrol.Add(vChecks); ?
< jl2012>
control.Add(vChecks); ?
< JeremyRubin>
yes
< gmaxwell>
jonasschnelli: it would but wumpus pointed out that that queue would potentially grow without bound if the client stops acking entirely.
< sipa>
jl2012: and the CScriptCheck objects have a pointer to the precomputation data
< JeremyRubin>
control.Add returns immediately
< wumpus>
yes, 100% reliable notification given any behavior of the receiver is not possible given finite space
< jonasschnelli>
gmaxwell: a queue limit is unavoidable... if you poll to lazy and the queue limit is to little, the only way to detect would be via the sequence number
< sipa>
jl2012: which means we must guarantee that that precomputation data remains alive as long as other threads may dereference the pointer
< wumpus>
some notification queue middleware logs events to disk in that case
< jonasschnelli>
wumpus: in long polling, the queue must be finite because we don't know when the client polls next,...
< wumpus>
e.g. rabbitmq etc, not by any means suggesting we take that up
< wumpus>
jonasschnelli: I agree there needs to be a limit realistically, it's enough if a client can detect missed events to resync
< jonasschnelli>
I personally think acking notifications is unnecessary,.. making it configurable seems also an overkill... so unsure.
< JeremyRubin>
jl2012: this is why I say a shared_ptr is what you really want, because the lifetime is automatically handled for you, whereas the vector is only by careful programming. However, that careful programming is currently correct :)
< sipa>
jonasschnelli: if you don't ask notifications, then your clients must manually deal with restarts... which implies having a way to ask for all events since some time... which, if you have it, removes the need for an event log entirely, as you can just use that all the time
< wumpus>
what's important is to document the limitations of the notification system in that regard
< sipa>
jl2012: do what?
< jl2012>
"I'd be a little more comfortable with this if PrecomputedTransactionData were passed around as const, so nobody's tempted to mess with "ready" for some crazy reason. But that can always be done later."
< gmaxwell>
shared_ptrs are a way to produce a software engineering disaster, because they let you be sloppy with ownership. There are cases of frestanding objects that don't have organized ownership but they are relatively rare. They also have non-trivial overhad.
< sipa>
jl2012: seems trivial?
< sipa>
i think you're overthinking it
< jl2012>
I'm thinking something like "const PrecomputedTransactionData txdata(tx);"
< sipa>
jl2012: cfields is only asking to pass it around as const; not to make the entire object const
< cfields>
^^ yep
< cfields>
though i think making the members const would be trivial too
< cfields>
either way, it's not really important for that PR. just an observation.
< jl2012>
it's even better if we could make the object const? There is no reason to make any modification
< JeremyRubin>
gmaxwell: am aware, but strictly speaking that is the point here: they are designed to track the actual needed lifetime, and are a 'tighter fit' to the problem in this situation. That they have problems is why I didn't PR my patch.
< cfields>
jl2012: just make the members const, then. use initializers rather than setting them in the function body
< JeremyRubin>
jl2012: I think that we shouldn't make it const, because there will be in the future per-tx state that we'll modify
< JeremyRubin>
e.g., if at some point in the future signature agg ends up needing a per-tx mutable struct that each input modifies
< gmaxwell>
JeremyRubin: in the case of the validation the ownership is just a straght line though, the creating code owns it, then the ownership is passed to the queue, then the ownership passes to the verifier thread..
< JeremyRubin>
Maybe it should be a different struct, but I think (to me), changing PreComputedTxData to PerTxData is straightforward.
< sipa>
JeremyRubin: i think that should be separate
< sipa>
if you want to avoid locking on the precomputed data
< JeremyRubin>
gmaxwell: yes, and then they are kept alive for longer than strictly needed. After the last scriptcheck executes for that tx, they can be freed.
< cfields>
agree. It'd be nice to keep the factual data separate from what's aggregated/stateful
< sipa>
JeremyRubin: yes, so? it doesn't change the worst case resource consumption
< JeremyRubin>
cfields: all the data should be factual in consensus?
< jonasschnelli>
sipa: persist the notification queue (with it's sequence numbers) to make it restart-safe?
< JeremyRubin>
sipa: I'm really only trying to talk about lifetimes here, which is the core of what jl2012 is discussing
< JeremyRubin>
sipa: asking
< sipa>
JeremyRubin: i'm just trying to point out that minimizing the time an object lives is not always wanted
< cfields>
JeremyRubin: i just meant that it can be safely computed ahead of time and seen as immutible after that.
< cfields>
factual was the wrong word, i suppose.
< JeremyRubin>
sipa: of course
< sipa>
jonasschnelli: that's one way (but it has pretty bad costs, if you want to make it durable - now you need synchronization across all files/databases)
< sipa>
jonasschnelli: another model is that you just have RPCs like listsinceblock, where the client tells the server effectively what it already knows
< sipa>
jonasschnelli: and then all you need is a notification like "there's something for you to look at"
< jonasschnelli>
sipa: by a rolling hash?
< sipa>
jonasschnelli: wut?
< sipa>
no, just like listsinceblock
< jonasschnelli>
how would the client tell the server what notfications it has... just the sequence number?
< jonasschnelli>
(need to check that)
< jonasschnelli>
thanks
< sipa>
no, i'm saying there are no notification
< sipa>
there are just state changes
< sipa>
and the client can ask "i've synced up to block X" or "i've seen transactions up to timestamp Y", or "the last balance update i saw was Z"
< sipa>
because most data has inherent sequencing anyway already
< jonasschnelli>
could it also do long polling? ... because IMO that is what users want (not const. polling)
< sipa>
yes, but the long poll just returns "there is something you should look at", not what
< jonasschnelli>
ah. And then the state update call.
< sipa>
... no
< sipa>
i mean *exactly* like listsinceblock
< jonasschnelli>
heh.. okay let me dive into there first
< sipa>
it has no sequence numbers, or notifications
< sipa>
it's a call where the client tells the server "hey, i've seen all transactions up to block X. what new transactions are there"
< sipa>
the next time the client calls, it uses the block hash at the time of its previous call
< sipa>
the server doesn't need to keep track of anything
< jonasschnelli>
would that also work with non block/tx data? Like new wallet txns (locally injected) or bandwith watermarks?
< jonasschnelli>
although not sure if there is a need for that
< sipa>
perhaps - maybe it's not possible for everything
< sipa>
bandwidth watermarks are easy... the client doesn't care if he misses one
< sipa>
for wallet txn you can use the txid of the last seen tx
< sipa>
jonasschnelli: my point is that you probably need something like that anyway, at least for when a new client starts up or went offline for a while
< jonasschnelli>
sipa: Yes. Your idea make sense.
< sipa>
and when you do, why bother with a separate event queue - all events could just be "hey something happened, you should check what"
< sipa>
that's a big concern i had with ZMQ... as it passes the actual new tx and blocks, but without guarantees that it delivers all
< sipa>
it was fixed with sequence numbers
< jonasschnelli>
maybe you should be able to not get such notifications on new mempool txns
< sipa>
but it could also have been fixed by not having it at all, and making clients query for the data they didn't have after every notifications"
< sipa>
anyway, just an idea
< jonasschnelli>
Thanks for sharing... need to think about it a bit more
< sipa>
having events you can subscribe to individually that persist etc... is certainly more convenient for clients
< sipa>
but it makes bitcoin core responsible for tracking who knows what, instead of leaving that up to clients (who are arguably in a better position to know what they already know)
< jonasschnelli>
Indeed. And also the ressources.
< sipa>
it would be pretty nice if there was a way to subscribe to "hey, let me know when txid X gets Y confirmations or gets reorged", but perhaps that could be done in a python script that just uses a simpler interface with bitcoidn
< achow101>
meeting?
< instagibbs>
yes
< sipa>
yes
< jonasschnelli>
#startmeeting
< lightningbot>
Meeting started Thu Oct 5 19:02:18 2017 UTC. The chair is jonasschnelli. Information about MeetBot at http://wiki.debian.org/MeetBot.
< gribble>
https://github.com/bitcoin/bitcoin/issues/8498 | Near-Bugfix: Optimization: Minimize the number of times it is checked that no money... by jtimon · Pull Request #8498 · bitcoin/bitcoin · GitHub
< meshcollider>
#11403 itself should be in there too probably?
< sipa>
wumpus: i haven't had the time to work further on 11403, though concept review is certainly welcome
< jnewbery>
I've been reviewing 11389 this afternoon. It looks generally good, but breaks assumevalid.py, which I'm trying to fix now
< jtimon>
s/locks/looks/
< BlueMatt>
sipa: I think we need a document on the various possible approaches, tbh
< sipa>
BlueMatt: yes, i'll work on that soon
< BlueMatt>
there are a few and talking through all of them is going to need something more formal
< BlueMatt>
thanks
< morcos>
achow101: does 10637 implement all the coin selection logic we discussed in SF or only BnB? Is there a high level description somewhere of what the PR is purporting to accomplish and what else will need to be done before 0.16?
< achow101>
morcos: only BnB
< achow101>
morcos: IIRC Murch is working on all of the coin selection stuff that we discussed
< wumpus>
btw I posted a proposed release schedule for 0.16.0 yesterday
< morcos>
achow101: ok.. i've already forgotten what that is, so might be nice to have that written up in an issue or something so we remember the goal and can think about how this BnB implementation is going to fit into the big picture
< achow101>
morcos: the description of what 10637 does is in the first comment.
< achow101>
I can make an issue for coin selection changes in general
< achow101>
*to keep track of
< wumpus>
ok, I think that concludes high priority for review proposals
< wumpus>
any other topics?
< achow101>
topic suggestion: bad block interrogation/invalid block peer banning
< wumpus>
#action achow101 make an issue for coin selection changes in general
< wumpus>
#topic bad block interrogation/invalid block peer banning
< achow101>
relevant PR is #11446 (I did this in class so it kinda sucks)
< achow101>
basically the idea is gmaxwell's. when we receive an invalid block, we want to make sure that all of our peers would also reject that block as invalid. If they don't ban them
< Murch>
I've been working on it, but since I do that in my free time in the evenings, it's been rather slow.
< luke-jr>
wumpus: this feels delayed?
< gmaxwell>
The general idea is that we aren't sufficiently agressive about punting peers on different consensus rules, so they can DOS attack us by sucking up slots, potentially hours per peer leaving us isolated... So there are number of things we can to do seek and destroy to speed up up.
< wumpus>
luke-jr: what feels delayed?
< luke-jr>
wumpus: 0.16
< Murch>
@achow101: If you want to collaborate on a write-up, I'd make myself available for that.
< gmaxwell>
luke-jr: release schedule is delayed because of 0.15.1
< achow101>
Murch: ok
< luke-jr>
i c
< wumpus>
luke-jr: yes, two months extra added, I mention that in the issue
< achow101>
what I wanted to discuss was the way to actually go about determining who to ban
< Murch>
@achow101: Gonna be traveling the next three weeks, so I might actually have more time. ;)
< sipa>
what is the issue with just looking at headers?
< gmaxwell>
achow101: I was kinda hoping we could implement something just from the messages we already get, it's my belief (could be wrong) that effectively we always learn the peers best header chain-- so we can begin kicking off peers based on that, as a first pass.
< luke-jr>
achow101: this contradicts the fixes in #10593
< gmaxwell>
achow101: I think we should be also drawing a distinction between inbound and outbound: the issue is what if we have a peer that accepts a broader set of blocks but would switch to our chain after learning of it.
< achow101>
gmaxwell: that's what I am not sure about. I don't think we necessarily know our peer's best header chain. suppose both us and them are fully synced, how do we know their best header chain until a new block appears?
< wumpus>
luke-jr: maybe two months is too much, but we'll see...
< luke-jr>
wumpus: nah, that sounds reasonable
< sipa>
achow101: when a new block appears, assuming it's PoW-valid to us, we'll learn about it through inv/headers/cb/...
< jonasschnelli>
Yes. +2 M seems okay to me
< gmaxwell>
sipa: but I believe he's right, we would have to wait for a new block, which is among the situations we're trying to resolve.
< achow101>
sipa: right, but I'm concerned about before a new block appears. we just connected to them or they just connected to us. we want to know then if we should ban them or not
< luke-jr>
IMO the desirable logic would be: for outbound connections, disconnect (don't ban) peers that aren't on the same chain; for inbound, tolerate it unless they reject a known-valid block
< sdaftuar>
we send getheaders messages on connect, typically
< gmaxwell>
For example say we are surrounded by ForkCoin peers, they are rejecting all bitcoin blocks. There are few forkcoin miners so they only get blocks once per day.
< gmaxwell>
We don't want to wait for them to get a new block just to figure out our current batch of peers are already on a chain we reject.
< achow101>
sdaftuar: are you sure? all I could find is that we sometimes send getheaders, not all the time
< sdaftuar>
achow101: we send getheaders messages to all our peers at some point after startup, but they might ignore them
< cfields>
sdaftuar: not to incoming light clients, i think?
< sdaftuar>
eg if they are doing ibd themselves or something
< sdaftuar>
not to light clients, correct
< sipa>
light clients don't matter here
< sdaftuar>
but to inbound node_network ndoes we do
< gmaxwell>
If we _always_ sent getheaders and then kicked outbound peers whos chain has a block we've rejected, then I think that is the best we can do per that concern (still not a perfect fix, since you're isolated until forkcoin finds at least one block)
< achow101>
sdaftuar: if we are sending getheaders, if they are on a different chain, we still wouldn't necessarily know because our start block may not be on their best chain
< gmaxwell>
oh hm. then perhaps we already do where it matters.
< sdaftuar>
gmaxwell: the difficult part might be that you don't know the chain they're on is invalid
< sdaftuar>
if it's got less work than yours
< RealM9>
Topic suggestion: s2x
< jonasschnelli>
RealM9: no
< luke-jr>
sdaftuar: do you care?
< luke-jr>
if they're rejecting your better chain, you want to disconnect them anyway
< gmaxwell>
sdaftuar: seems like a seperate concern, we should also be kicking outbound peers that have less work than us, I think.
< RealM9>
Ok, but community is pretty interested. Are you going to change POW?
< sdaftuar>
gmaxwell: i think that would be a good idea, yeah
< gmaxwell>
But it would be silly to be overly agressive.
< sipa>
RealM9: us?
< achow101>
sdaftuar: gmaxwell what I propose is that we send a getheaders for our earliest known invalid block (within a certain time frame) and see if they respond with invalid blocks
< luke-jr>
RealM9: that's a decision for the community, not for developers. anyhow, ask on #bitcoin if you really want to discuss it
< gmaxwell>
achow101: I don't think we need to do that: for sync purposes any outbound peer we should be makign sure we learn their headers chain period (they may have a better chain than us and we should sync up ASAP)
< sdaftuar>
achow101: i'm not sure that's necessary?
< gmaxwell>
achow101: if we're already doing that we'll notice known invalid block in their header chaip (well we will once we have code for that)
< sdaftuar>
i think if we do gmaxwell's suggestion of booting inbound peers who are on less work chains, then we'd be in good shape
< sdaftuar>
s/inbound/outbound/
< luke-jr>
I think we may actually want to track the headers of invalid chains..
< achow101>
what about inbound peers?
< gmaxwell>
For _inbound_ I think we should be setting a flag that they're consensus inconsistent which excludes them from the inbound peer management connection reservation.
< sdaftuar>
achow101: i think we should more aggerssively evict inbound peers if they appear to be on invalid chains
< gmaxwell>
so they'll get kicked off in favor of other inbound peers.
< meshcollider>
Agreed
< gmaxwell>
so we don't need to be agressive: they'll just get pushed out by other inbound peers.
< luke-jr>
consider: if an invalid chain has higher hashrate than the real chain, and then suddenly the invalid chain's hashrate drops off, without an equivalent increase on the main chain, we should consider that a possible attack and hold back on confirming transactions until it is resolved
< sdaftuar>
gmaxwell: yes i agree with you
< achow101>
my point is how do we know that an inbound peer is on an invalid chain?
< gmaxwell>
luke-jr: I think there is some need for smarter wallet confirmation logic but I think thats a seperate matter. (there was a paper 6-ish months ago that also points out the the reorg probablity math in the whitepaper is somewhat incomplete)
< sdaftuar>
achow101: set a flag if they relay an invalid block/blockheader
< luke-jr>
gmaxwell: right, but this is relevant because we can't assume "relays invalid headers" means the other node *accepts* the invalid block
< gmaxwell>
and we still interogate their headers if they're NODE_NETWORK/NODE_LIMITED
< luke-jr>
sdaftuar: we intentionally relay blocks before checking validity now
< achow101>
sdaftuar: that requires them to have a block to relay to us, which could take hours or days
< gmaxwell>
luke-jr: the protocol does not have you realying a header of a block you haven't accepted. If you do that you risk dos attacking peers already.
< sdaftuar>
achow101: i don't think we need to worry as much about inbound peers
< sdaftuar>
achow101: for instance an attacker can already try to use all your inbound slots and not send you anything
< gmaxwell>
luke-jr: the only place that happens in the protocol is HB BIP152 messages.
< achow101>
sdaftuar: right, ok
< luke-jr>
gmaxwell: which may be all you see from CB peers
< gmaxwell>
sdaftuar: yes, for inbound we can just deprive them of reservations.
< sdaftuar>
luke-jr: even with bip152 the headers need to be valid
< luke-jr>
sdaftuar: yes, the header itself; but it can be a valid header for an invalid block
< gmaxwell>
yes, though we'd catch it on the _next_ block.
< sdaftuar>
luke-jr: if it builds on an invalid chain, i believe the header would be invalid
< achow101>
so when we connect to an outbound peer, we will send them a getheaders so we know their best headers chain and ban accordingly
< luke-jr>
(note I tried to keep track of peer bestblocks in #10512 and basically gave up)
< gmaxwell>
when they relay a CB message for a child of an invalid block.
< gmaxwell>
achow101: yes, but based on the above I believe we already always send it.
< achow101>
the other part of 11446 is to ban other peers for relaying us an invalid block for which we already know is invalid
< gmaxwell>
achow101: because we send it to nodenetwork peers and outbound always are (or or disconnected)
< achow101>
but I'm not sure how that interacts with compact blocks
< gmaxwell>
achow101: FWIW, I think we should probably be just disconnecting and not banning.
< sdaftuar>
achow101: oh that interaction might be tricky
< achow101>
gmaxwell: why not ban?
< gmaxwell>
I think the interaction isn't too bad, for this purpose a BIP152 CB HB block is relaying you the header of its parent.
< luke-jr>
achow101: in a softfork, old nodes will send invalid blocks
< luke-jr>
potentially
< gmaxwell>
achow101: because it's hardly any better and it means that when some dimbulb tries running forkcoin it results in him being unable to run Bitcoin (perhaps concurrently) on the same host.
< achow101>
gmaxwell: ok
< gmaxwell>
it also blocks inbound from that peer, which we'd be find allowing.
< gmaxwell>
s/find/fine/
< gmaxwell>
In general we should be moving away from bans except when the thing we banned for was expensive for us.
< achow101>
so 11446 can really just be reduced to an ~1 line change to disconnect on a header for a block we already know is invalid
< BlueMatt>
yea, that
< sdaftuar>
achow101: agree, though we have to be careful about compact blocks i think
< sipa>
topic suggestion: dealing with platform-specific code
< jonasschnelli>
luke-jr: I can continue to work on 11383 if you want?
< jonasschnelli>
(remove the auth stuff :P)
< luke-jr>
jnewbery: certainly not that simple.. still need to resolve wallet name to CWallet earlier
< jnewbery>
ok, well I've got a branch that works with just that change. Happy to share with you
< gmaxwell>
Sounds good.
< luke-jr>
jnewbery: push it and I'll take a look
< jnewbery>
thanks
< wumpus>
#topic dealing with platform-specific code (sipa)
< sipa>
i've recently been looking into faster parallel hashing code
< wumpus>
hashing as in sha256?
< sipa>
in particular, for 8-way parallel SHA256 (which would be useful in merkle root computation and block deserialization), a 5x speedup is doable with AVX2
< sipa>
and maybe 2.5x with SSE2
< wumpus>
and parallel in this case means computing multiple hashes of multiple pieces of data at once?
< sipa>
correct
< luke-jr>
how much speedup is this for the entire IBD?
< gmaxwell>
luke-jr: It saves something like 10 minutes on IBD. But the greater impact is in block relay.
< wumpus>
(I guess there are constraints there, do all the inputs need to be the same size?)
< luke-jr>
I imagine merkle root is a tiny fraction of the overall process, but otoh it's also possibly a blocker on parallelization
< gmaxwell>
Where hash tree computation is most of the time.
< luke-jr>
gmaxwell: it is? :O
< sipa>
wumpus: yes and no; for now, it's just a primitive that you give a pointer to N 64-byte inputs, and produces 32-byte outputs
< luke-jr>
oh, because the signature checks are cached?
< BlueMatt>
in terms of compact block relay, merkle root calculation and deserialize are about the only big timesinks before you can relay
< sipa>
wumpus: which is specific for merkle root computation
< gmaxwell>
luke-jr: yes for HB BIP152 we don't to validation except hashing!
< sipa>
but it can certainly be adapted
< wumpus>
sipa: ok
< cfields>
sipa: i had a scare when reviewing some new leveldb crc32 changes that i thought (at first glance) could be a consensus issue. I was very angry at myself at that point for not adding a fallback un-optimized verification of the optimized path.
< sipa>
anyway, there are multiple ways to integrate this: separate asm code, inline asm blocks, or code using intrinsics (my preference, it's much more easy to review, and has no OS-specific warts like the L label prefix...)
< gmaxwell>
sipa has actually implemented the 8-way AVX2 sha2 and a hash tree computation that uses it... along with specialized implementation of 64-byte input double sha2.. which affords an addition 20%-ish speedup over generic sha2.
< cfields>
very cool :)
< wumpus>
I prefer intrinsics
< wumpus>
(except for arm32 whre they suck)
< sipa>
so, for intrinsics... do we want to have a separate LIBCRYPTO_AVX2 LIBCRYPTO_SSE2 LIBCRYPTO_... with different compile flags each?
< gmaxwell>
Historically, For some code you cannot achieve equivilent performance w/ intrinsics because you must manage register allocation precisely for things to work, but that isn't the case here....
< sipa>
or could we rely on __attribute__((target("avx2")))
< wumpus>
but 64 bit platforms the SIMD instructions have been specially tweaked to work well with compilers and intrinsics
< sipa>
(which works on both clang and gcc)
< cfields>
sipa: i think we should test for the target attribute and use it if possible, but not completely rely on it
< cfields>
iirc that improves dispatching time as well?
< sipa>
cfields: no
< wumpus>
different compile flags for different compile units, that's the only portable way
< luke-jr>
sipa: I think we can't assume intrinsics for all platforms, so we want the separate lib route
< gmaxwell>
dispatching is via a function pointer ultimately in all those cases.
< wumpus>
luke-jr: you're confusing, that's not about intrinsics
< sipa>
the only difference is avoiding the need for build system complication
< wumpus>
intrinsics inthis case are headers like xmmintr.h which provides functions that work on vector types
< sipa>
exactly
< luke-jr>
wumpus: __attribute__((target("avx2"))) isn't an option for separate asm code, though, right?
< sipa>
luke-jr: it also isn't needed for asm code
< cfields>
gmaxwell: isn't there elf data that allows them to be setup at load time?
< wumpus>
oh no no ELF magic please
< luke-jr>
hmm
< cfields>
not by hand, i thought __attribute__(target) did that behind the scenes
< sipa>
target("avx2") just means "this function is compiled as if -mavx2 was passed on the command line
< sipa>
cfields: GCC also has target("default"), where you can have multiple versions of the same function... which causes automatic dispatch to be added
< sipa>
that's non-portable and has other issues
< gmaxwell>
cfields: they're setup at load time, yes-- but they're still just a function pointer, which we could also have setup at load time. Though it is nice that the function override trick can make them run before main.
< wumpus>
I'm normally not scared of low level ELF hacking, but for bitcoin, let's try to keep it safe and portable
< luke-jr>
sipa: how does it behave if I have an explicit -mno-avx2?
< sipa>
(in particular clang doesn't have that)
< cfields>
sipa: ah yes, that's what i was thinking of.
< sipa>
cfields: so i'm not suggesting using that
< sipa>
luke-jr: i assume it overrides it
< cfields>
ok
< luke-jr>
I suppose I can test it
< wumpus>
yes, gcc can do it automatically on some platforms, but I'm afraid the only portable way is to make our own dispatch logic
< sipa>
yes, we'll want our own dispatch logic anyway
< wumpus>
we already have some CPUID bits checking
< sipa>
so we can test things
< wumpus>
so it's nothing new erally
< sipa>
and report which version is being chosen
< cfields>
np, i wasn't suggesting. just trying to understand the advantages of one vs the other.
< wumpus>
yes, exactly
< sipa>
but if possible i'd like to avoid the overhead of needing half a dozen libcrypto_XXX.a things that need to be linked in everywhere
< sipa>
though that's really the only advantage
< wumpus>
so I'd say: yes, use intrinsics instead of inline/offline asm, and use our own dispatching, and compile units compiled with appropriate compiler flags
< sipa>
okay.
< gmaxwell>
can we say prefer intrinsics instead of use? :) I don't think we'd eschew inline asm if we thought it was better in a particular case.
< wumpus>
yes regarding build system it's just verbose, not really complex
< cfields>
sipa: see my point above about a fallback, though. In the case of mismatch hashes, i think it's worthwhile to re-check with a generic implementation before deciding it's failed.
< gmaxwell>
cfields: we should be testing these things in startup tests.
< luke-jr>
(yes, it seems to override -mno-*
< wumpus>
gmaxwell: well if there's a case you can do much better than the compiler, sure...
< * BlueMatt>
has a weak preference for compile units, but only cause I'd use them in FIBRE for my FEC stuff, too, but thats not much of a reasoning
< * luke-jr>
hopes we can have POWER9 asm in 0.16 <.<
< * BlueMatt>
agrees
< cfields>
gmaxwell: an implementation bug in some branch of one optimized path is scary...
< gmaxwell>
cfields: try differently if it fails is just not reasonable in a lot of cases; and often would add a lot of complexity (now you have to not cache hashes, but instead only use hash-verify methods) ... and we don't have expected values for thigns like single transaction hashes, just hash roots.
< cfields>
gmaxwell: in particular, the crc issue had to do with incoming data alignment on x86_64
< wumpus>
cfields: I agree
< wumpus>
cfields: I think we should only do asm optimization in cases where it really makes a lot of difference, for that reason, ther risk has to be worth it
< gmaxwell>
cfields: yes, thats something that always needs careful review and we should have unit tests that also stress alignment.
< wumpus>
special-casing everything makes things a lot harder to review, and test, especially when it starts to need different kinds of hardware
< wumpus>
but for testable low-level primitives like SHA256 I'd say it's ok
< gmaxwell>
good thing no one is talking about special casing everything. :)
< gmaxwell>
yea, sha2 etc have simple testable interfaces.
< wumpus>
no, that's just one extreme, I've seen soome graphics drivers which are scary in that regard :)
< gmaxwell>
But benchmarks!
< wumpus>
oh let's special case 4x4 tiles, 4x5 tiles, 4x6 tiles, ... for 3 different architectures
< wumpus>
right :)
< cfields>
mmm. I don't see the harm in doing a quick re-check in a few certain cases (merkle mismatch is a good example)
< wumpus>
special-casing benchmarks is a curious form of over-learning
< cfields>
anyway, i've made my case
< gmaxwell>
cfields: because it requires restructing the code to not return hashes but instead only have uncachable hash_Verify methods.
< wumpus>
cfields: re-check in what case?
< luke-jr>
although someone did manage to screw up xpub serialisation at one point IIRC
< wumpus>
cfields: you mean re-run the validation w/ different implementations if an incoming block fails?
< wumpus>
(what about false positives?)
< cfields>
wumpus: that's a big hammer, but yes-ish
< gmaxwell>
cfields: and for small functions like a hash a check in an innerloop will measurably lower performance. ... and you also create the opposite problem, what if the alternative function is the wrong one?
< gmaxwell>
(I'd actually consider whole block level more reasonable)
< wumpus>
gmaxwell: I think he means on a high level
< wumpus>
on the inner level it's just NASA-level crazy, let's run three implementations and see which ones agree
< sipa>
i think re-checking a block if it fails is reasonable... but why switch hash functions? it's massively more likely your CPU is fried than that the hash function implementation is wrong all along and you never noticed
< gmaxwell>
but then the dispatch is mutable not just set once at init. :(
< wumpus>
yeah ... I think we're overdesigning this
< gmaxwell>
right we have a constant slow stream of complaints from users whos hosts have falsely rejected the blockchain.
< wumpus>
just continue with what you were doing sipa :)
< cfields>
gmaxwell: just have a generic non-dispatchable one
< gmaxwell>
I would like to see that improved somehow.
< wumpus>
any other topics?
< wumpus>
oh wait, it's time
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Oct 5 20:01:57 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< gmaxwell>
sneaky way to learn developers mailing addresses.
< sipa>
(way too WIP to PR)
< achow101>
ooh free t-shirts
< * sipa>
found the student
< meshcollider>
gmaxwell: Heh true
< cfields>
sipa: very cool. didn't mean to rain on your parade.
< luke-jr>
gmaxwell: never got any spam that I can tell came from last year's
< luke-jr>
just don't use your home address for mailing address ;)
< wumpus>
sipa: nice
< gmaxwell>
cfields: so imagine that sha2 implementation isn't alignment safe. you get a block and miscompute one of the hashes due to alignment. ... I don't see any way of efficiently accomplishing your 'try another function' approach that would stop the false rejection.
< sipa>
wumpus: too bad the 64-byte specialized generic-x86 code is slower than the generic-data-size SSE4-specialized version
< sipa>
wumpus: otherwise i'd straight up PR the 64-byte specialized code
< wumpus>
the avx code looks very recognizable
< sipa>
wumpus: it's pretty much search replace on SSE4 code
< wumpus>
heh
< gmaxwell>
it's only detectable at the hash root check at the end a long computation pipeline... when that fails do we just go back and re-deseralize the entire block with different code to compute new hashes in the CTransactions?
< gmaxwell>
(and if we do, we magnify a DOS vector for someone sending us invalid blocks, though perhaps not enough to worry about)
< wumpus>
that's another advantage of intrinsics, it's usually easier to review than straight up asm
< sipa>
wumpus: absolutely
< wumpus>
no need to keep track in your head where all the registers go
< cfields>
gmaxwell: addmittedly that's ugly, but yes, i think that's worth considering
< sipa>
which may be a good thing or a bad thing
< sipa>
1) no need to keep in your head where the registers go!
< sipa>
2) no way to tell the compiler to keep a certainly value in a register!
< wumpus>
well combined with the pipelining/interleaving that optimized code needs, and the large number of registers that SIMD architectures have, that can be quite difficult
< gmaxwell>
I wish you could assign variables to registers, and have the compiler yell at you if you tried to assign two live variables to the same register.
< wumpus>
usually there's so many registers that it would be good to be able to tell that something is *not* worth storing in a register
< sipa>
i believe there is actually a way (in GCC) to force a particular variable into a particular register
< sipa>
int x asm("%edx");
< gmaxwell>
cfields: and still doesn't help with accepting something we shouldn't, which is usually a more serious issue.
< gmaxwell>
cfields: if we had some generic infrastructure to retry a failed validation (e.g. to cope with lossy hardware) then perhaps what you're thinking about could be dropped into it.
< cfields>
gmaxwell: i'm not saying it's something we have to do, or something that wouldn't be ugly. I'm moreso coming from a place where I was in full-out panic for a few hours because I thought newer x86_64 machines were about to start diverging...
< morcos>
I also don't want to rain on anyone's parade, and this is OSS so people work on what they find interesting, but I think we shoudl be careful to think about what are priorities for the project
< morcos>
More importantly however, we shoudl be careful about making changes that are not easily reviewable by more than 1 or 2 people unless they are really warranted
< morcos>
I've been thinking more about this since bech32. I think bech32 is completely awesome, I'm super happy we're doing it and imo its a good priority project. But it essentially got no review. Did anyone other than sdaftuar review it?
< morcos>
Sometimes things will have to be like that, but it shoudl be a tradeoff we consider carefully... how tricky are we trying to be vs how much is it warranted
< * BlueMatt>
tends to agree, though noting that a part of my agreement is my different priorities from some others - performance is maybe much more of a concern for many others in the project more than I
< morcos>
I haven't evaluated that in the context of the parallelized hashing, but its something I think we should
< gmaxwell>
morcos: bech32 had more review then it appeared because we solicited extensive review before publishing the bip.. what didn't get review was the checksum itself other than myself and pieter, until sdaftuar. ... but who reviewed the prior address checksum?
< gmaxwell>
Clearly no one, because it needlessly sucked. :P
< sipa>
well it was too late to review it by the time any of us were around
< luke-jr>
I didn't review Bech32 because I figured it was over my head (especially magic checksum stuff).
< gmaxwell>
There were many design changes to the earliest Bech32 proposal that arose out of review, e.g. the delimiter character.
< morcos>
Yes and of course the 2 people that can't be blamed are you and pieter since you did the work. But I don't want us to fall into a trap of just assuming if something is too difficult or outside of our field to review properly that someone else must be doign a good job with it
< morcos>
gmaxwell: and yes i was referring to the checksum
< morcos>
but thats a good point...
< gmaxwell>
We deal with this for libsecp256k1 too, that fact that it's in a different repo is just enabling you to ignore it. :)
< gmaxwell>
Though we do have effective review there too.
< gmaxwell>
Though not as much as I'd like.
< sipa>
but at least for secp256k1 it's clear what the goal is (implement secp256k1 EC correctly), so someone could review e.g. just the test and judge that they're sufficient for that goal
< cfields>
morcos: yes, well said. I think that's why I find the asm changes scary. I spent a full day trying to learn and understand the sse42 optimized sha2, and only because it was failing some tests. At best, I agree that it looks right, but I could never say it with any degree of certainty. I deferred to "it passes all the tests".
< sipa>
with bech32, the effort is in the design, not the implementation, and there is relatively little proof that the design actually has the properties it claims to have
< gmaxwell>
But thats the same thing as reviewing the bech32 checksum. and fwiw, if feedback on bech32 we got was we needed more reviewers for the checksum, I would have gone and asked a libsecp256k1 contributor to do it.
< gmaxwell>
because I know e.g. andrew (or peter dettman) aren't frightened off by math.
< morcos>
I think my feedback is one meta level up. There should have been more people that questioned how much reivew it got
< gmaxwell>
yes, agreed.
< gmaxwell>
well I was thinking that before you commented. Started thinking it as soon as I saw other bitcoin software had merged it.
< morcos>
so i know i'm not going to review the hashing stuff, just want to make sure other peopel are going to ensure it is properly reviewed
< morcos>
i'm frightened off by code. :)
< gmaxwell>
had someone commented on that earlier, I would have agreed. I reviewed the checksum work too, but pieter and I worked a lot togeather on it, and if he screwed up he probably would have tained me.
< sipa>
the cool thing about hash functions is that they have essentially no branches... so it's very hard (though not impossible, see the alignment issue) to make it be incorrect only for a hard-to-detect small subset of inputs
< gmaxwell>
yes on hash function correctness, but that doesn't help with BECH32 design, which I guess as your point above. It's easy to be confident a new implementation of it is conformant... not so much that the design is good.
< sipa>
right
< gmaxwell>
and still no one has basically reviewed our decision to use a cyclic code -- perhaps if we were call coding theorists someone would have known an even better tool... but there is a limit to how far down a rabbit hole we can go.
< morcos>
gmaxwell: but part of my point was the tradeoff on how important the feature is. i am definitely +1 on bech32. and not negative on parallel hashing just raising points for us to think about
< morcos>
anyway, got to run
< BlueMatt>
someone wanna close 11454?
< gmaxwell>
morcos: I think you can decompose your concerns into two folks-- people will work on whatever they like, but if it's going to get merged it needs to deserve the required review attention, since that isn't just an indivigual decision.
< gmaxwell>
forks*
< gmaxwell>
morcos: and then unrelated, we shouldn't be in a state where people don't review or even provide meta review of things which are two mathy or too low level and so they assume that they won't be of use.
< gmaxwell>
And the thing to encourage there is that even if its over your head you can still ask some of the right meta questions.