< sipa>
oh, no; you're just talking about the sanity check
< sipa>
that should be an assert really, and it can go above, indeed
< jimpo>
sipa: So it's safe if I change that to an assert at the top?
< sipa>
jimpo: imo, yes
< bitcoin-git>
[bitcoin] jimpo opened pull request #11411: script: Change SignatureHash input index check to an assert. (master...sighash-bounds-check) https://github.com/bitcoin/bitcoin/pull/11411
< jimpo>
I'm running the test suite on it now. Let's see...
< bitcoin-git>
bitcoin/master 3826253 Wladimir J. van der Laan: rpc: Handle `getinfo` locally in bitcoin-cli w/ `-getinfo`...
< bitcoin-git>
bitcoin/master 5e69a43 John Newbery: Add test for bitcoin-cli -getinfo...
< bitcoin-git>
bitcoin/master c9a4aa8 Wladimir J. van der Laan: Merge #10871: Handle getinfo in bitcoin-cli w/ -getinfo (revival of #8843)...
< bitcoin-git>
[bitcoin] laanwj closed pull request #10871: Handle getinfo in bitcoin-cli w/ -getinfo (revival of #8843) (master...cli-getinfo) https://github.com/bitcoin/bitcoin/pull/10871
< bitcoin-git>
[bitcoin] kallewoof opened pull request #11413: [wallet] [rpc] sendtoaddress: Add explicit feerate option to sendtoaddress (master...explicit-fee) https://github.com/bitcoin/bitcoin/pull/11413
< ThomasV>
can someone clarify the syntax of importmulti?
< ThomasV>
"Array of strings giving pubkeys that must occur in the output or redeemscript"
< ThomasV>
does that mean that for each execution path of redeemScript, you have to import a different array of pubkeys?
< sipa>
ThomasV: signing with anything other than pure multisig is not supported, so there can't be multiple execution paths
< sipa>
so no, it's just the list of all public keys that appear
< ThomasV>
sipa: will the syntax need to be changed for other scripts?
< sipa>
i think it's also only necessary for single-pubkey cases
< sipa>
if you give the redeemscript, you're good
< sipa>
if you want solvability or signability
< sipa>
ThomasV: the idea is you give a) what script/address you're interested in b) what you want for it (watching, solving, signing) c) all the information necessary for that (redeemscripts, public keys, private keys, ...)
< ThomasV>
sipa: so if you pass pubkeys you do not need to pass redeemscript?
< sipa>
rather the other way around
< sipa>
passing pubkeys (i think) is only useful for single-key cases (pay to pubkey etc)
< ThomasV>
but why is it an array, then?
< ThomasV>
and the doc says "Array of strings giving pubkeys that must occur in the output or redeemscript"
< ThomasV>
that suggests an execution path
< sipa>
yes - you're free to pass in multiple keys if they all appear
< sipa>
that doesn't mean that you need to
< ThomasV>
hmm
< sipa>
presumably it's for being future proofness
< ThomasV>
ok, so future plan is to extend that to arbitrary scripts?
< sipa>
we'll only ever be able to sign the scripts we can sign
< sipa>
importmulti is just a way for passing in all the information the signing logic may possibly need
< gmaxwell>
I don't believe its computationally tractable to be able to sign actually arbritary scripts.
< ThomasV>
sipa: the signing logic needs to know the position of the signature in the witness. this position may differ depending on the execution path
< sipa>
ThomasV: if it knows the redeemscript, it knows all possible execution paths
< ThomasV>
so I would expect that info to be passed to importmulti
< gmaxwell>
Though there are presumably large classes that could be signed automatically, or partially using the fill in the blanks technique.
< sipa>
(but currently no scripts with conditionals are supported in the signer)
< ThomasV>
sipa: it depends what you mean by "it knows". does the signing logic execute the redeem script in order to figure that out?
< ThomasV>
(or parse it)
< sipa>
it uses template matching
< sipa>
again, the current logic can't deal with anything other than P2PK, P2PKH, P2SH, and simple multisig
< ThomasV>
ok, so the only template it currently has is pure multisig
< sipa>
however, i believe that the interface for importmulti doesn't need changing
< sipa>
if the signing logic gets extended
< sipa>
say we'd add support for signing an A-and-B OR C-and-D like script, if you've given it the redeemscript (which includes the 4 pubkeys), it can theoretically sign for something like that
< ThomasV>
gmaxwell: signing arbitrary scripts is not tractable if you expect the signing logic to understand the script. however, I think that burden could be moved to the user who owns the privkey
< ThomasV>
when you import a key, you could also tell the wallet in which context it should be used
< gmaxwell>
I think you're thinking of what I mean by fill in the blanks technique, where you attempt to verify the script, assume all checksigs pass and record their inputs, and then go and sign any that you know the keys for.
< gmaxwell>
or perhaps you didn't. :)
< ThomasV>
I mean to tell the wallet which execution path is the key for
< sipa>
i don't understand why that's necessary, if the wallet knows all execution paths anyway
< ThomasV>
the wallet does not know how the script is going to be executed
< sipa>
so?
< ThomasV>
so, if I sign a transaction, I should provide that info
< sipa>
usually there are just a few branches, it can easily try all of that
< sipa>
but in general - i'm sure there are cases where it's not feasible (say you have 30 independent conditionals - you'd need to try 2^30 combinations)
< ThomasV>
sipa: there might be branches that you do not want to sign.
< gmaxwell>
ThomasV: I don't follow, it's for the execution path(s) that involve the corresponding public key.
< sipa>
at some point you can't expect a generic wallet to deal with arbitrarily complex scenarios, and you'll just need to write your own software i think
< ThomasV>
gmaxwell: maybe the same pubkey occurs in 2 branches
< gmaxwell>
Then it's the same secret... which choice you were signing with would be a signing time decision, I think, not an importmulti-time decision.
< ThomasV>
gmaxwell: exactly. my point is that making it an import time decision makes signing arbitrary scripts a lot simpler.
< ThomasV>
you only need to pass the position of the signature in the witness, and the length of the witness
< ThomasV>
and you don't need to fill the blanks, or parse scripts
< gmaxwell>
I don't follow your thinking at all there.
< sipa>
if there is a script that has a pubkey in multiple branches, meaning you're able to participate in multiple branches, and you have that key... why would you - at importing time - want to be forced to choose only one of those branches?
< sipa>
sure, you may want to make that choice at signing time
< gmaxwell>
that sounds like you're thinking signatures in the witness always have a fixed position? they don't.. okay but I understand why you wouldn't want any understanding of script in general.
< sipa>
with signature aggregation there will just be a single signature anyway :)
< sipa>
much easier! *ducks*
< ThomasV>
gmaxwell: my point is precisely that they do not have a fixed position. that's what I mean by role
< gmaxwell>
but since a "stick signature in this position" approach cannot sign arbritary scripts or even remotely arbritary, is it really that unreasonable to do simple template matching of the script?
< ThomasV>
gmaxwell: why can't it sign arbitrary scripts?
< ThomasV>
the logic is "if I see that redeemscript, then I sign"
< gmaxwell>
ThomasV: because the position your signature needs to go into can depend on data set by another signer (for example) in each specific instance of signing.
< ThomasV>
no template matching needed
< ThomasV>
gmaxwell: in that case you import the same key with several roles
< gmaxwell>
e.g. if you have a signature that has a policy that looks like (A || (B && C)) && D and you are D, the stack element your will sign with depends on which of the paths are in use by the OR people.
< gmaxwell>
ThomasV: well that would be a very inefficient interface if there were many paths, I don't think it would make sense for us to do it that way, since we can simply just run through the script and see what is actually needed.
< gmaxwell>
Esp with recently proposed future enancements e.g. mark or jl's tree constructions, there could be a huhdred possible locations for a redeemscript.
< sipa>
i'm sure there could be an alternate more lowlevel interface, where you say "this redeemscript can be signed for with this witness template... with a signature-for-key-A in position this, and a signature-for-key-B in position this, ..."
< sipa>
but that question isn't even relevant at this point - we only do signing of specific templates anyway
< sipa>
which i think will cover the vast majority of use cases - especially the uses cases for those who aren't able to write custom signing code themselves
< gmaxwell>
well the fill in the blanks would cover a far greater subset of those...
< sipa>
with MAST-like constructions something like that could make sense - where you're able to sign for a MAST root even if you don't know all MAST branches - only the ones you participate in
< ThomasV>
gmaxwell: good point. indeed, that would be inefficient.
< ThomasV>
sipa: so, the witness of a partially signed transaction would be a tree? in greg's example: ((sigA (sigB, sigC)), sigD), where sigX is either a signature or an empty placeholder
< ThomasV>
until it's fully signed
< sipa>
ThomasV: have you seen achow101'w partially signed transaction format?
< bitcoin-git>
bitcoin/master f77f0e4 Andrew Chow: Add warnings field to getblockchaininfo
< bitcoin-git>
bitcoin/master 8502b20 Andrew Chow: Unify help text for GetWarnings output in get*info RPCs
< bitcoin-git>
bitcoin/master 395cef7 Andrew Chow: Change getmininginfo errors field to warnings...
< bitcoin-git>
[bitcoin] laanwj closed pull request #10858: [RPC] Add "errors" field to getblockchaininfo and unify "errors" field in get*info RPCs (master...getblockchaininfo-errors) https://github.com/bitcoin/bitcoin/pull/10858
< jonasschnelli>
BlueMatt: how did you know I'm working on the RPC long poll notifications?
< jonasschnelli>
I started to pick it up yesterday
< BlueMatt>
lol, I'm just going through old prs
< jtimon>
I'm going to miss the meeting, review beg #8498 and #8994 if at any moment seems relevant
< gribble>
https://github.com/bitcoin/bitcoin/issues/8498 | Near-Bugfix: Optimization: Minimize the number of times it is checked that no money... by jtimon · Pull Request #8498 · bitcoin/bitcoin · GitHub
< wumpus>
luke-jr: yes, you could ignore DST for yourself and live in GMT time, the difficulty is shop opening times and such :)
< jnewbery>
hi
< kanzure>
hi.
< Chris_Stewart_5>
hello
< wumpus>
oh no the bot is missing
< wumpus>
#action find the bot
< achow101>
is it?
< petertodd>
wumpus: one of the perils of replacing humans with machines
< wumpus>
ehh.. no it isn't, sorry
< wumpus>
petertodd: but it lives in the cloud! humans can't live in a cloud!
< wumpus>
#topic High priority for review
< * BlueMatt>
's is staying....
< BlueMatt>
again
< BlueMatt>
10663 got merged
< BlueMatt>
same with 10871
< jonasschnelli>
hi
< wumpus>
yes, finally
< BlueMatt>
=D
< wumpus>
BlueMatt: yours has wallet in the name, that seems to put a curse on it with regard to reviewers
< luke-jr>
XD
< jonasschnelli>
Maybe someone can review #10387 (in high prio), ideally cfields
< gribble>
https://github.com/bitcoin/bitcoin/issues/10387 | Implement BIP159, define and signal NODE_NETWORK_LIMITED (pruned peers) by jonasschnelli · Pull Request #10387 · bitcoin/bitcoin · GitHub
< jonasschnelli>
Would be great to have this in 0.16
< wumpus>
which reminds me
< cfields>
jonasschnelli: will do asap
< wumpus>
#topic put up 0.16 release schedule
< jonasschnelli>
cfields: thanks!
< BlueMatt>
wumpus: yea, I think so...I havent been pushing it cause folks want 0.15.1 first, but afterwards I really want that to go in asap, cause I really want to have it in early in 0.16 so it simmers for months
< wumpus>
10387 is already in "high priority for review"
< jonasschnelli>
Yes
< wumpus>
BlueMatt: seems it has had quite some review though
< jonasschnelli>
Just no reviews. :)
< BlueMatt>
#11089 needs to get removed from high priority, it needs major rebase
< gmaxwell>
jonasschnelli: Oh, I'm surprised that patch is so large, I was kinda hoping to see it widely backported so pruned nodes on older branches could see it advertised.
< jonasschnelli>
Maybe next to the release schedule for 0.16, ... what do we want in 0.16?
< sipa>
most of 11089's functionality is included in #11403
< jonasschnelli>
gmaxwell: I'm happy if you see a way how to reduce it
< jonasschnelli>
Or split it
< gmaxwell>
jonasschnelli: yea, will look. Sorry, didn't see it until now.
< wumpus>
11089 removed for now
< jonasschnelli>
Although +264 −13 shouldn't be too large
< jonasschnelli>
(incl. new test)
< morcos>
oh man i feel like i'm buried under PR's to review ... maybe i'd make some progress if you could review via tweet
< jonasschnelli>
heh
< luke-jr>
would be nice to have GUI multiwallet in 0.16, but #10615 is stuck on desires for refactoring :/
< gribble>
https://github.com/bitcoin/bitcoin/issues/10615 | RPC: Allow rpcauth configs to specify a 4th parameter naming a specific wallet (multiwallet RPC support) by luke-jr · Pull Request #10615 · bitcoin/bitcoin · GitHub
< BlueMatt>
luke-jr: I think it is unacceptable to add so many more ifdef WALLET in bitcoin server
< jonasschnelli>
luke-jr: can you remove the rpcauth part? It's not necesdarry and contraversial.
< wumpus>
this is tagged for 0.16 right now (both issues and PRs):https://github.com/bitcoin/bitcoin/milestones/0.16.0
< BlueMatt>
(and believe it would be easy to remove them)
< gmaxwell>
jonasschnelli: I don't understand why the flags set would change at runtime? Setting the flag is a property of the node's configuration.
< sipa>
morcos: well there is @BitcoinMerges...
< gmaxwell>
(and in our case we don't have any configuration where it shouldn't be set, AFAIK)
< wumpus>
morcos: well at least we can fit twice as much code in a tweet now...
< morcos>
sipa: that's brilliant, at least it would constantly remind me
< jonasschnelli>
gmaxwell: maybe no longer required with a single NODE_NETWORK_LIMITED bit
< luke-jr>
jonasschnelli: I think that's possible, but I don't see why it should be controversial, and doesn't help with BlueMatt's refactoring wants
< jonasschnelli>
luke-jr: it's a fact that it is contraversial
< jonasschnelli>
* controversial
< gmaxwell>
jonasschnelli: ah! right, so this code was originally based on the more complex spec.
< BlueMatt>
luke-jr: I dont think I'm the only one who objects to ifdef wallet's being added to http server cpp files........
< jonasschnelli>
gmaxwell: thanks for pointing out... i'll review it and check if I can remove the runtime switching
< wumpus>
BlueMatt: no, you aren't, that's obviously wrong, we've been working hard to remove those as they cause circular refs :(
< jnewbery>
luke-jr: cherry-picking the non-controversial commits from your PR gives us the GUI functionality without adding a bunch of server->wallet dependencies or the rpcauth stuff
< luke-jr>
jnewbery: I don't see how that's possible
< gmaxwell>
jonasschnelli: my completely ignorant (only barely read the patch) thought is that there should be one commit which should be a nearly one line change to just set the bit everywhere.. then another commit or two for the relay, then another for selection. And the first commit we'd backport, maybe the second, but not the third.
< luke-jr>
BlueMatt: so httprpc calls a callback for "populate JSONRequest with metadata" which is only used for this one thing?
< luke-jr>
BlueMatt: or what did you have in mind?
< wumpus>
see e.g. #7965. We want to go toward 0 wallet references in bitcoin_server
< jonasschnelli>
gmaxwell: I thought it is basically like that (factored in a bit more commits though)
< achow101>
wumpus: we can check getinfo off that list now
< wumpus>
achow101: thanks good point
< BlueMatt>
luke-jr: yes, that would be sufficient
< BlueMatt>
imo
< luke-jr>
ok, I'll try that
< wumpus>
achow101: updated
< BlueMatt>
luke-jr: you may wish to seek concept acks on the *idea* of rpc auth-based multiwallet
< BlueMatt>
I'm ok with it, but I dont know if others are
< luke-jr>
#action refactor #10615 for a "populate JSONRequest" callback, and separate out rpcauth stuff
< gribble>
https://github.com/bitcoin/bitcoin/issues/10615 | RPC: Allow rpcauth configs to specify a 4th parameter naming a specific wallet (multiwallet RPC support) by luke-jr · Pull Request #10615 · bitcoin/bitcoin · GitHub
< gmaxwell>
jonasschnelli: ah, right though the setter isn't simple like I expected but now I understand why.
< jonasschnelli>
BlueMatt , luke-jr : the rpcauth multiwallet is completely orthogonal to GUI multiwallet
< gmaxwell>
and that should be fixable.
< wumpus>
jonasschnelli: indeed, let's not confound things unnecessarily
< luke-jr>
jonasschnelli: yeah, I'll split it off
< jonasschnelli>
yes. It's smells like smuggling in controversial stuff packed in a nice feature
< gmaxwell>
Yes, please split those things up. (I say this as a supporter of rpcauth based multiwallet control)
< sipa>
i'm ok with rpcauth-based multiwallet, but i think it may be a longer-term thing; it would probably require some review of security of non-wallet RPCs (should a wallet-specific rpc user be allowed to call stop?)
< luke-jr>
sipa: well, it wasn't meant to be a security thing, but maybe we can consider that
< sipa>
yeah, just saying - i think multiwallet gui can be done much faster
< luke-jr>
right
< BlueMatt>
I was assuming rpcauth-based multiwallet would only allow you to call wallet functions
< sipa>
everyone clearly wants it
< gmaxwell>
sipa: main interest I have in it is anti-footgun. Can't spend accidentally what you don't have access too.
< jonasschnelli>
rpcauth opens up the usecase and suddenly users think Core is a 100k multiuser wallet system
< jonasschnelli>
we just need to be sure we want this
< wumpus>
yes, multiwallet GUI isn't controversial in any way
< gmaxwell>
jonasschnelli: Then we can simply say it isn't. I don't think that is a good argument.
< morcos>
yeah lets pleast punt on rpcauth for now... seems better use of our time to move towards splitting off wallet
< wumpus>
the gui even has some scaffolding already to be able to handle multiple wallets IIRC, of course it's not complete
< achow101>
gmaxwell: people will probably still use it that way though
< luke-jr>
leaving `stop` etc accessible makes it clearly not that IMO
< jonasschnelli>
We may end up in the same situation like the accounts... i'm not against it. But we should discuss it wisely and not related to GUI multiwallet
< wumpus>
morcos: +1, I'm kind of sick of discussing that tbh
< jonasschnelli>
`/rpcauth (ack)
< luke-jr>
wumpus: #11383 is there already, just needs to get past this initial stuff
< BlueMatt>
jonasschnelli: please give a tl;dr; of what it is :)
< wumpus>
jonasschnelli: no, no problems with it, I think it's superior to other ways we've considered
< jonasschnelli>
RPC long poll would allow similar push benefits then ZMQ offers but without any dependencies
< gmaxwell>
jonasschnelli: long poll for what specifically?
< luke-jr>
jonasschnelli: we already have RPC long poll
< wumpus>
for any of the current notifications
< wumpus>
blocks, transactions, wallet events
< gmaxwell>
you mean like blocks and transactions...
< jonasschnelli>
you open an http connection (with a timeout of lets say 60seconds)... once a new block/tx pops in, you get it back and reopen the timeout 60 channel
< gmaxwell>
ah.
< jonasschnelli>
It's how Cellphone OSes do push notifications
< wumpus>
yes. the stuff that now requires launching an external notify script
< jonasschnelli>
very nat friendlly
< wumpus>
jonasschnelli: yes, it's very basic and easy to use
< gmaxwell>
Well. "Better than ZMQ"-- part of the argument against it in the past was socket monopolization but with libevent now that should be much less of an issue.
< jonasschnelli>
Very easy to use... no ZMQ library, just an http client
< sdaftuar>
how does reopening work? i don't understand how you avoid missing transactions in between
< luke-jr>
gmaxwell: hopefully. libevent has a history of having problems in this area IIRC
< jonasschnelli>
sdaftuar: you just open the http channel again... there is a queue on the server..
< wumpus>
it's not meant as a competition to ZMQ, ZMQ will stay
< jonasschnelli>
losing notifications is impossible
< jonasschnelli>
(or very unlikely)
< luke-jr>
sdaftuar: you have a session
< gmaxwell>
jonasschnelli: impossible and very unlikely are not the same. :P
< cfields>
didn't we have this discussion wrt waitfornewblock and friends? I'm missing how this is different
< wumpus>
zmq is for low latency notifications, longpoll is for easy notifications over the same channel RPC already works
< jonasschnelli>
yeah.. probably same level then ZMQ (where you can also loose nots)
< wumpus>
zmq requires careful extra setup
< luke-jr>
or order the events, and have the client request with the last-seen-event (as GBT does)
< jonasschnelli>
It would allow clients to better interect with Core without opening an ZMQ & RPC channel (only RPC)
< gmaxwell>
if you are using it to notice new transactions, losing a message means you fail to credit a customer with a $10,000 deposit... and look like a theif or even actually lose the money. :)
< jonasschnelli>
and it would allow *auth based notifications"
< jonasschnelli>
like wallet notifications
< jonasschnelli>
(which should be behing the auth)
< jonasschnelli>
notifications may lead to an RPC roundtrip...
< wumpus>
gmaxwell: true, but the -Xnotify that launches a script can also fail to launch a script in some cases, losing notification
< BlueMatt>
if its for wallet you can just use it to get info about when you need to call listsinceblock
< gmaxwell>
ZMQ has wire protocol incompatiblity between versions, which makes it harder to use except in very tightly coupled setups.
< jonasschnelli>
It just ellimitens constant triggering
< BlueMatt>
which isnt so bad...but for mempool its unclear how you can do it safely
< wumpus>
gmaxwell: we have to be realistic too
< sipa>
wumpus: but on the next notification you'll notice it
< gmaxwell>
wumpus: Is there a reason we can't give a reliable notification?
< wumpus>
sipa: yes, that's why the zmq notifications have sequence numebrs too
< luke-jr>
we probably don't want to use the same LP for wallet and non-wallet
< wumpus>
sipa: you'll notice that you've lost an event, you just don't know what it is so need to do reconciliation in that case
< jonasschnelli>
My LP PR (which I'm currently re-doing) has sequence numbers
< cfields>
ah
< wumpus>
gmaxwell: well the problem is that send buffers can be full
< jonasschnelli>
If you miss a notification, do a data re-sync
< sipa>
i wonder if we shouldn't think about something more generic... you configure "i have an RPC client, which is interested in all events which satisfy condition X, Y, Z....", and then there can be means for the client to read the event log, or get notifications when there are new events, ...
< wumpus>
gmaxwell: 100%(ish) reliable notifications means storing them to disk if the client isn't processing them
< sipa>
and the server keeps track of what the client has seen and hasn't
< BlueMatt>
(as long as all of those events come out of the CValidationInterface =D)
< luke-jr>
sipa: that makes me think of pruning - "I need to use block data, so don't prune until I say I'm done with it"
< wumpus>
gmaxwell: I mean reliable messaging/notification is a huge topic in itself..
< luke-jr>
sipa: not the same thing, but perhaps has some overlap
< gmaxwell>
wumpus: or allowing the send buffer to grow to the point that it gets swapped out and ultimately crashes. :)
< sipa>
luke-jr: indeed, it seems related
< wumpus>
gmaxwell: yes, in which case it's lost too
< jonasschnelli>
sipa: my LP PR does basically something like that. You can register a UUID for notifications and the server keeps track what you have received and what not
< jonasschnelli>
It's multi-client capable
< wumpus>
luke-jr: yeah there's certainly overlap there
< gmaxwell>
wumpus: yes, though cold start init is something that everything has to have and has to have tested, as it happens at every start.
< wumpus>
luke-jr: block consumers ideally should be able to set a low-water mark
< luke-jr>
maybe they should share jonasschnelli's UUID
< wumpus>
gmaxwell: well it needs reconciliation, you could also do that if you miss a notification ( which can be detected from the sequence number)
< wumpus>
gmaxwell: so if you have something to handle potential crashes, you also have something to handle the case where a message gets lost
< luke-jr>
although for block consumption, we'd want to store it in a persistent db IMO
< luke-jr>
(or could use rwconf and have a config param, but that seems ugly for this)
< gmaxwell>
obligitory question: Is any user asking for this or is it just interesting?
< jonasschnelli>
gmaxwell: Yes. Me as an user... :)
< wumpus>
yes yes yes
< luke-jr>
gmaxwell: the block consumption part would allow Armory to do pruning, at least ;0
< wumpus>
joinmarket for example would like a better notification mechanism
< jonasschnelli>
(for the hardware wallet client i'm working)
< cfields>
luke-jr: hmm, so clients could issue a single rpc command and have new blocks announcements replayed after a day offline?
< wumpus>
gmaxwell: they currently have a hack where you have to set walletnotify in bitcoin.conf, which isn't very flexible and easy to forget
< luke-jr>
cfields: I don't see why not
< wumpus>
gmaxwell: if their software could just subscribe to events through RPC, that problem'd be solved
< BlueMatt>
jonasschnelli: if its not a big bother, then, as wumpus points out, can you ask the joinmarket guys if your proposed protocol is sufficient for them? would be good to be in sync on it
< luke-jr>
related: my GUI NetWatch branch has a circular memory-efficient buffer of weak-linked txs and blocks
< wumpus>
gmaxwell: because polling sucks
< jonasschnelli>
BlueMatt: good point
< wumpus>
it causes extra delays, at least, and uses more CPU time
< wumpus>
(depending on how fast you poll)
< jonasschnelli>
wumpus: and the UX may suck,... LP gives you a TX, block within ms not s
< wumpus>
I really don't understand why we're arguing whether to add a sane notification system at all and considering repetitive polling good enough
< gmaxwell>
ISTM the caller will need a consierable amount of complexity to avoid losing data, vs sending a poll every coupld seconds. Polling faster than seconds hardly makes sense in the context of bitcoin in any case, since txn take seconds to propagate through the network.
< gmaxwell>
wumpus: because we already have one almost unused unreliable notification system.
< wumpus>
sigh, never mind
< wumpus>
another topic?
< jonasschnelli>
sipa had one
< gmaxwell>
bleh, I'm not trying to bludgon it.
< wumpus>
and no, zmq is not unused
< jonasschnelli>
IMO it is heavely used in some enterprises
< wumpus>
just that you're not using or it's not good/reliable enough for you it doesn't mean no one is using it
< gmaxwell>
I've never encountered someone using it in production. I don't doubt that it is somewhere.
< wumpus>
as I've said many times, loss of messages can be handled if it can be detected, and it can be detected using the sequence numbers
< * BlueMatt>
's preferred method of notification is the "hey, you should poll now" method
< sipa>
i think there is a lot of merit in creating a reliable event logging system for clients to observe
< luke-jr>
BlueMatt: indeed, blocknotify is sane IMO
< sipa>
if we have that, adding a notification for "you have new events!" seems both trivial and obvious
< jnewbery>
suggested topic: next Bitcoin Core meetup
< jonasschnelli>
BlueMatt: ZMQ / LP is "hey, you should poll now"
< cfields>
agreed. and agree with wumpus. Seems like a no-brainer to me.
< luke-jr>
jnewbery: another one?
< jonasschnelli>
We only send hashes
< BlueMatt>
jonasschnelli: I wouldnt even send hashes
< gmaxwell>
But we don't get a report flow on it that suggests that its in use... And the advice I give is to poll because it's reliable, hard to get wrong, and has no scaling issues on any system that can keep up with the network. And the reason I was asking questions is because I was trying to understand what the gain in having another one would be... one thought I had was maybe it would be more reliab
< BlueMatt>
just do the old "disconnect upon new data" pattern
< sipa>
19:19:49 < sipa> i'd like to bring up segwit wallet support
< wumpus>
gmaxwell: you can do polling in addition to processing notifications, that's what I mean with 'reconciliation'
< morcos>
notification system for topic suggestions?
< gmaxwell>
Obviously if people want to work on it, I don't mind. I'll even spend some time reviewing it. But there are so many other things we've left half complete... so thats just my reservation. I'm sorry for frustrating you.
< wumpus>
anyhow, time for sipa's topic
< wumpus>
#topic segwit wallet support
< sipa>
yay!
< morcos>
for the record i'm also wary about adding more and more notification systems, but that's not to say i don't think we shouldn't improve from where we are now
< wumpus>
morcos: you've just missed that topic
< wumpus>
:p
< sipa>
so, i have a PR #11403 which i think implements most of what we want, except some import/export and message signing
< sipa>
however, i think there are 2 differences compared to what we discussed before
< sipa>
1) importprivkey works, and actually imports all corresponding addresses for that given key (P2PKH, P2SH-P2WPKH, P2WPKH)
< wumpus>
nice
< sipa>
that's not how i want things to work long term, but it's very hard and probably confusing to users to almost everywhere follow the "a key corresponds to all 3 forms of corresponding addresses", except when importing (and importing is inevitably needed anyway for testing)
< sipa>
2) once you generate a segwit address with this patch, you can't really downgrade to older software anymore (unless you go fix your missing addresses with addwitnessaddress), though no new backup is needed
< luke-jr>
why 2?
< BlueMatt>
wait, why?
< morcos>
2 confuses me
< sipa>
which differs from the earlier idea of every new addresses effectively calling addwitnessaddress
< BlueMatt>
yea, what we discussed was auto-addwitnessaddress
< morcos>
i think it depends on what we are expecting 0.16 to do
< sipa>
well, auto-addwitnessaddress doesn't actually achieve what you want either
< morcos>
if the manual upgrade of a wallet to 0.16 is going to auto add all 3 versions of all keys, then what you are suggesting seems maybe ok... but that will lead to more bloat in 0.16 as opposed to just having the scripts already in the wallet moved over
< sipa>
effectively what the patch does is just implicitly making redeemscripts for all your keys known, without writing them to the file
< meshcollider>
Oh so if you wrote them to the file downgrading would be fine
< achow101>
sipa: why not write them?
< sipa>
meshcollider: not really... new transactions you receive while downgrading are risky
< sipa>
achow101: bloat, and it doesn't fully solve the problem
< BlueMatt>
sipa: yes, why did you decide to do that over the previous discussion?
< luke-jr>
morcos: don't add segwit stuff for old keys..
< sipa>
BlueMatt: because keys the old version adds to the keypool during auto-topup won't get their witnesses added, and will go undetected
< gmaxwell>
meshcollider: writing them doesn't make downgrading fine though, because the downgraded one won't be adding them for things in the keypool. so in the presences of restores they'd potentially silently lose funds.
< luke-jr>
would be pretty ugly if the address list tripled in size simply with an upgrade; ideally we should only list one type per key, perhaps excepting the import case
< BlueMatt>
sipa: huh? I figured youd "auto-addwitnessaddress" when you getnewaddress/get the address for gui/etc
< BlueMatt>
not when the key is generated
< sipa>
BlueMatt: that won't work at all
< gmaxwell>
luke-jr: we're kinda stuck with that right now, obviously when we change the wallet design we can move to a model where there is a 1:1 matching between chains and keys. But thats a major redesign.
< sipa>
BlueMatt: as old versions will then miss all transactions received while offline
< luke-jr>
gmaxwell: if you getnewaddress with an older version, you won't get a segwit address, so they don't *need* to be added..
< BlueMatt>
sipa: yes, please explain why
< sipa>
(their keypool entries won't have witnesses, so won't be watched for)
< luke-jr>
gmaxwell: we can leave existing keys alone, at least
< BlueMatt>
ah, i see your point
< gmaxwell>
BlueMatt: then backup and restore is completely no durable and you'll miss payments if you start with a restored wallet...
< gmaxwell>
yea.
< luke-jr>
gmaxwell: only if you restore a backup with an old version?
< luke-jr>
that seems like a very niche problem area
< sipa>
there is a possible best-effort approach of adding witnesses for all keypool entries and new addresses... but given that that's not even water tight...
< luke-jr>
"when restoring a backup, don't use old versions" seems the straightforward "fix"
< gmaxwell>
luke-jr: no, the best we could do is store a flag where the segwittyness starts and handle that, but because users are _already_ using addwitnessaddress this would not resolve their current recovery problems while adding for all does.
< gmaxwell>
luke-jr: this whole discussion is about old restores, if you don't do that all is fantastic in sipa's code.
< luke-jr>
gmaxwell: so I'm a user. I upgrade, and suddenly my address list has 500 new addresses. So instead of making new ones, I just use those up..
< BlueMatt>
sipa: so downgrades are broken anyway cause they wont know to scan for witness-ified scripts anyway
< BlueMatt>
(and you have to do a rescan or so)
< gmaxwell>
by old restores I mean downgrade.
< luke-jr>
gmaxwell: it sounds like sipa's code will fail to work properly if you simply downgrade with a wallet that's used a segwit address
< luke-jr>
no backup/restore involved
< BlueMatt>
sipa: so it sounds like we're back to needing to bump wallet version?
< sipa>
BlueMatt: that would be nice, yes...
< gmaxwell>
luke-jr: right, it doesn't downgrade.
< gmaxwell>
luke-jr: but downgrading is just not possible to support here. At best it can look kinda like it works but lose funds.
< BlueMatt>
sipa: so are we back to the wallet-overhaul-ish approach, then?
< morcos>
sipa: it would be really helpful if you would spell this all out in a document. including how it'll interact with 0.16
< luke-jr>
if it doesn't downgrade, I see no value in this temporary approach
< sipa>
morcos: it's spelled out
< sipa>
BlueMatt: NO
< gmaxwell>
jesus fucking christ.
< sipa>
BlueMatt: all we need is a version number
< morcos>
lets find flaws in the overall plan now, and not wait and realize we made a mistake now that makes 0.16 more difficult
< sipa>
overhauling will take far longer
< morcos>
sipa: where is it spelled out
< sipa>
morcos: it doesn't make 0.16 harder
< gmaxwell>
luke-jr: If you don't see value in supporting segwit this year then I don't have anything more to discuss with you.
< morcos>
i agree, overhaul is goign to take a long time
< morcos>
but i just want to understand now what we are going to do
< BlueMatt>
sipa: well at a minimum its now blocked on hd upgrade, then, no?
< sipa>
BlueMatt: no
< BlueMatt>
why not?
< gmaxwell>
BlueMatt: there is a pretty straight forward fix, set the version to maximum and introduce a new version field.
< BlueMatt>
argh, lets not
< sipa>
hd upgrade requires a new backup
< sipa>
_this doesn't_
< BlueMatt>
hd upgrade requires no more backup than keypool topup
< sipa>
it should simply be "minimum software version to read this file is X"
< BlueMatt>
(hd upgrade doesnt neccessarily imply you *must* wipe your keypool)
< sipa>
BlueMatt: it will write a hd master key though, which must be backed up
< BlueMatt>
(and is also quite simple)
< BlueMatt>
sipa: no more than topping up your keypool? or you mean you'd want to be able to use segwit-wallet without keypool topup and with an encrypted (and locked) wallet?
< sipa>
so to be clear... it *is* possible to make downgrade work, if restoring a backup with an older version isn't a concern - though at the cost of some bloat
< sipa>
and it may make expectations unclear
< BlueMatt>
you mean as long as you dont restore a backup, just use the latest wallet, right?
< BlueMatt>
ie via the addwitnessaddress method?
< sipa>
yes
< gmaxwell>
restoring a backup is always a concern though. I don't understand why downgrading suddenly shows up as a hard requirement. It's something that you generally can't do with a new feature except through the cheat of introducing it first but disabled.
< BlueMatt>
gmaxwell: yes, but we generally "support" it in the sense that you have to manually do -walletupgrade if you want new features
< sdaftuar>
gmaxwell: i agree that restoring a backup is important, and supporting downgrade not a big deal
< morcos>
sipa: i'm sorry if i missed this, but can you point me to where this is spelled out or just explain what we will do when you upgrade to a 0.16 wallet? Add 3x number of keys pkscripts to your ismine set? which we'll do for all keys in your wallet for all time?
< gmaxwell>
BlueMatt: don't restore a backup, don't run concurrent copies... perhaps some other corner cases we haven't considered.
< BlueMatt>
and thus we support running the latest version and going back to the previous release eg if there's some critical issue for you
< sipa>
> Every wallet key implicitly adds its corresponding P2WPKH script to the wallet's known redeemscripts - without writing to the file. This is simpler, needs significantly less space on disk, needs no one-time upgrade for existing keys, but does mean that once a SegWit address has been used, you can't really downgrade to older software anymore.
< BlueMatt>
"don't restore a backup"???
< sipa>
^ from my PR description
< BlueMatt>
sipa: yes, to me that implies we need to bump wallet version
< morcos>
i would distinguish that from the addwitness approach where you wouldn't have to do that (but you could have an i'm upgrading from a backup option that does do that to make sure you didn't miss anything)
< BlueMatt>
which is fine, but that leads us back to the question of hd upgrade
< BlueMatt>
we can do some hack to let people upgrade to segwit-wallet without an hd upgrade
< BlueMatt>
or we can jsut implement hd upgrade, which I think is rather trivial
< gmaxwell>
BlueMatt: Trivial but invalidates their backups.
< instagibbs>
promag, meeting wait please
< BlueMatt>
gmaxwell: see previous discussion on keypool? or do you mean locked wallets?
< BlueMatt>
thats a rather narrow use-case, no?
< sipa>
BlueMatt: yeah, i don't know
< sipa>
i may be convinced of doing the addwitnessaddress approach
< * BlueMatt>
would prefer hd-upgrade over dirty hacks, but I'm open to discussion
< sipa>
though i'm uneasy with the inability to restore while downgrading
< BlueMatt>
yea, I'd tend to agree
< luke-jr>
what if we bump the wallet version only in backups?
< sipa>
on the other hand - everything will be fixed by upgrading again and rescanning
< gmaxwell>
BlueMatt: hdupgrade invalidates backups, even without a locked wallet. Though thats another point you have to unlock to do it.
< wumpus>
3 minutes to go
< gmaxwell>
I think if restore isn't reliable we shouldn't run with the older version. Thats a big footgun in a mixed wallet where you may not notice a non-trivial percentage of funds vanishing.
< luke-jr>
backups don't need to use the same version as the latest-wallet
< sipa>
luke-jr: people use cp to make backups
< luke-jr>
sipa: that's broken
< luke-jr>
:/
< sipa>
luke-jr: irrelevant - it will cause lost funds
< luke-jr>
it can cause lost funds even as-is
< gmaxwell>
luke-jr: yes, but it's a minor addition to make it not load in older software.
< jnewbery>
suggested topic: next Bitcoin Core meetup
< sipa>
so, open for discussion perhaps on the PR: auto-add witnesses so downgrading when not restoring a backup works, or some scheme of versioning to make old software fail
< sipa>
endtopic
< wumpus>
#topic suggested topic: next Bitcoin Core meetup (jnewbery)
< BlueMatt>
sdaftuar: points out that downgrade + restore is always broken
< achow101>
jnewbery: already?
< BlueMatt>
cause even if you bump the version number....
< instagibbs>
NYC :D
< BlueMatt>
hence why its nice to have an explicit walletupgrade
< jnewbery>
ok, just a quick announcement - we're in the early stages of planning the next one in NYC the week of March 5th 2018
< sipa>
jnewbery: what timeframe were you thinking of?
< luke-jr>
BlueMatt: can you elaborate on sdaftuar's point after meeting?
< jnewbery>
more details to follow by email
< BlueMatt>
luke-jr: yes
< wumpus>
achow101: better to plan it long in advance so people can take it into account, instead of on short notice
< luke-jr>
I'm expecting a baby in February, so I will probably pass on the March meetup
< jonasschnelli>
jnewbery: thanks for the date! thanks for organising.
< cfields>
jnewbery: woohoo! thanks for planning!
< sipa>
jnewbery: cool, on the way back from Financial Crypto :)
< instagibbs>
\o/ marked on it calender
< * jonasschnelli>
can't attend NYC... to bad
< wumpus>
jnewbery: works for me
< cfields>
luke-jr: didn't know that, congrats :)
< wumpus>
jonasschnelli: that's two times US in a row, let's plan another one in europe next
< luke-jr>
cfields: thx
< promag>
europe +1
< jonasschnelli>
wumpus: I can organise fall 2018 in Europe
< luke-jr>
we haven't done Australia or Russia yet. but those might just be inconvenient for too many people :p
< jnewbery>
New York is almost in Europe. Just a short flight :)
< meshcollider>
Australia +1 ;)
< luke-jr>
jnewbery: lol
< achow101>
lol
< jonasschnelli>
jnewbery: hehe
< wumpus>
austrialia would be fine with me too, russia meh
< instagibbs>
not incredibly bad for Tokyo folks
< * cfields>
votes for fanquake to host one at his place
< wumpus>
anyhow thanks everyone, it's time to wrap up the meeting
< * sipa>
votes for Iceland in the summer
< jonasschnelli>
Iceland would be cool...
< jonasschnelli>
and cold
< wumpus>
jonasschnelli: finally a meeting in UTC+0!
< meshcollider>
Oops didn't mean to post that at a snippet lol
< jnewbery>
sipa: is there a write-up of your current plans for the wallet in 0.15.0.1 and 0.16? I saw the PR notes, but I feel like I don't fully understand the longer term plans and the concerns that were raised in the meeting
< luke-jr>
meshcollider: eh, people can still review/open PRs without meeting discussion ;)
< cfields>
jonasschnelli: got a few min to talk about 10387?
< wumpus>
yes, meetings certainly aren't intended to become a bottleneck for discussion
< sipa>
jnewbery: sounds like something i need to write up
< instagibbs>
+1
< wumpus>
IRC meetings are basically for topics that need as many people as possible to give input
< wumpus>
for the rest please just discuss on github or IRC outside the meeting...
< sipa>
jonasschnelli: iceland can be up to 20C in summer :)
< jnewbery>
sipa: thanks :)
< meshcollider>
Yeah although the PRs Luke linked were in another repo and obviously super controversial, not just review heh
< meshcollider>
And yeah sipa: a writeup would be nice yeah I'm still confused as well haha
< sipa>
jonasschnelli: ... if you're lucky
< wraithm>
24 hours of sun heats things up (sort of)
< promag>
sipa: is bench/bech32.cpp useful?
< sipa>
promag: it's just for addresses... hardly performance critical
< promag>
theres is base58, for comparison?
< gmaxwell>
there is bench base58 because someone wanted to make 'optimizations' that sounded reasonable, but it's not reasonable to optimize things without a benchmark. :)
< gmaxwell>
if someone wanted to 'optimize' bech32 though they're just rip out the C++ one and put in the C one. :P
< gmaxwell>
personally I'd just suggest waiting until someone wants to optimize it, though that code was already pessimized for the sake of readability, and the C version is quite a bit faster.
< promag>
why not add the C version?
< sipa>
it's less readable
< sipa>
we prefer C++ish code over Cish code where possible
< gmaxwell>
I don't think there is any case where the performance matters here.
< gmaxwell>
or at least where a factor of two would matter.
< promag>
it's also the reference implementation
< sipa>
there's also a C++ reference implementation
< sipa>
(which the code in my PR is based on)
< promag>
just curious.. anyway I've added bench/bech32.cpp while reviewing #11167