< bitcoin-git> [bitcoin] tjps opened pull request #10059: [trivial] Removed one Boost usage and headers in util.cpp (master...tjps_boost) https://github.com/bitcoin/bitcoin/pull/10059
< bitcoin-git> [bitcoin] jnewbery closed pull request #10055: [test] [POC] Stop and start bitcoind nodes asynchronously in functional tests (master...stopstartasync) https://github.com/bitcoin/bitcoin/pull/10055
< bitcoin-git> [bitcoin] achow101 opened pull request #10060: [RPC] Ensure an item exists on the rpcconsole stack before adding (master...fix-rpcconsole-empty-stack) https://github.com/bitcoin/bitcoin/pull/10060
< bitcoin-git> [bitcoin] tjps opened pull request #10061: [net] Added SetSocketNoDelay() utility function (master...tjps_nodelay) https://github.com/bitcoin/bitcoin/pull/10061
< bitcoin-git> [bitcoin] kallewoof opened pull request #10062: [net] Clean up the net.h file. (master...20170322-cleanup-net) https://github.com/bitcoin/bitcoin/pull/10062
< bitcoin-git> [bitcoin] jonasschnelli pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/02d64bd929c9...86f7d5b69bb7
< bitcoin-git> bitcoin/master 97b8213 practicalswift: Fix parameter naming inconsistencies between .h and .cpp files...
< bitcoin-git> bitcoin/master 86f7d5b Jonas Schnelli: Merge #10029: Fix parameter naming inconsistencies between .h and .cpp files...
< bitcoin-git> [bitcoin] jonasschnelli closed pull request #10029: Fix parameter naming inconsistencies between .h and .cpp files (master...fix-header-parameter-naming-inconsistencies) https://github.com/bitcoin/bitcoin/pull/10029
< bitcoin-git> [bitcoin] jonasschnelli pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/86f7d5b69bb7...7b585cf70ec5
< bitcoin-git> bitcoin/master c4a6929 Matt Corallo: Clarify assumptions made about when BlockCheck is called
< bitcoin-git> bitcoin/master 7b585cf Jonas Schnelli: Merge #9558: Clarify assumptions made about when BlockCheck is called...
< bitcoin-git> [bitcoin] jonasschnelli closed pull request #9558: Clarify assumptions made about when BlockCheck is called (master...2017-01-blockcheckeddocs) https://github.com/bitcoin/bitcoin/pull/9558
< bitcoin-git> [bitcoin] jonasschnelli pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7b585cf70ec5...3568b30ca31d
< bitcoin-git> bitcoin/master 6d8fe35 Andrew Chow: 'help' rpc commands autocomplete...
< bitcoin-git> bitcoin/master 3568b30 Jonas Schnelli: Merge #9500: [Qt][RPC] Autocomplete commands for 'help' command in debug console...
< bitcoin-git> [bitcoin] jonasschnelli closed pull request #9500: [Qt][RPC] Autocomplete commands for 'help' command in debug console (master...help-rpc-autocomplete) https://github.com/bitcoin/bitcoin/pull/9500
< bitcoin-git> [bitcoin] MarcoFalke pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/3568b30ca31d...dfef6b6af080
< bitcoin-git> bitcoin/master 72163d4 practicalswift: [tests] Remove unused and duplicate imports
< bitcoin-git> bitcoin/master 3897459 practicalswift: [tests] Remove unused variables
< bitcoin-git> bitcoin/master dfef6b6 MarcoFalke: Merge #10047: [tests] Remove unused variables and imports...
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #10047: [tests] Remove unused variables and imports (master...remove-unused-python-imports) https://github.com/bitcoin/bitcoin/pull/10047
< bitcoin-git> [bitcoin] MarcoFalke pushed 6 new commits to master: https://github.com/bitcoin/bitcoin/compare/dfef6b6af080...a230b0588788
< bitcoin-git> bitcoin/master 94b528b Russell Yanofsky: [qa] Remove bumpfee.py get_change_address hack
< bitcoin-git> bitcoin/master e6b2963 Russell Yanofsky: [qa] Get rid of nondeterminism in bumpfee.py...
< bitcoin-git> bitcoin/master 1dfd64f Russell Yanofsky: [qa] Make bumpfee.py test function order consistent...
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #9701: Make bumpfee tests less fragile (master...pr/bumpfee-fragile) https://github.com/bitcoin/bitcoin/pull/9701
< luke-jr> woo, sizefp code builds finally
< luke-jr> although it's still incomplete.. need a good way to detect the "no full size proofs on right sides of duplicate merkle link hashes"
< luke-jr> but I'd better get to bed so I can be up for the meeting :x
< btcdrak> wow it's Thursday again already... how time flies
< bitcoin-git> [bitcoin] flack opened pull request #10063: add missing spaces so that markdown recognizes headline (master...patch-1) https://github.com/bitcoin/bitcoin/pull/10063
< gmaxwell> meeting in 20 minutes.
< petertodd> sipa: IIRC there's something not unlike that in rust with deref coercion
< gmaxwell> luke-jr: does your prover optimize that it's cheaper to provide merkle membership proofs when you show transactions with common parents in the tree? e.g. Say that the sizes are 1000 900 100 910 you get more size-proof per proof-size if you select 1000,900 instead of 1000,910.
< luke-jr> gmaxwell: the current code doesn't
< gmaxwell> luke-jr: I would eliminate transactions by first assuming all are in and then asking "can I leave out this leaf?" for each interior node and child, then remove the one that eliminates the most proof size per transaction size proven.
< gmaxwell> (proof size including the membership proofs, and the extra data, so it'll tend to remove txn that have a lot of extradata)
< gmaxwell> luke-jr: this is another reason for a softfork increasing the minimum transaction size in the future. Your proofs should also be able to prove that there is a transaction smaller than a minimum.
< gmaxwell> would be interesting to profile your proofs on the existing chain with a fake block limit of 750k and also see how the proof size changes as a function of the minimum transaction size.
< sipa> PING
< gmaxwell> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier
< kanzure> hi.
< jonasschnelli> hi
< petertodd> hi
< wumpus> #startmeeting
< sipa> hi
< lightningbot> Meeting started Thu Mar 23 19:00:20 2017 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
< achow101> hi
< gmaxwell> Welcome to meeting.
< cfields> hi
< btcdrak> greetings!
< wumpus> proposed topics?
< btcdrak> holiday at the beach?
< petertodd> btcdrak: that's what the financial crypto conference is for
< gmaxwell> wumpus: The segwit address proposal on the list.
< petertodd> gmaxwell: +1
< wumpus> sounds good to me
< btcdrak> +1
< wumpus> #topic segwit address proposal
< jonasschnelli> proposed topic: DER encoded priv keys in wallet
< jonasschnelli> ack #sw addr format
< sipa> general opinions/questions about the segwit addr format?
< gmaxwell> We might have made a strategic error in getting 1:1 comments from too many people, causing a darth of comments on the list. Comments on the list would be good even if they're just "I reviewed this before, LGTM"
< jonasschnelli> Andreas S. concern about QRCode?
< kanzure> who has reviewed bech32 by now?
< sipa> jonasschnelli: petertodd responded, by i can give some extra commemtd
< petertodd> gmaxwell: along thsoe lines, I think making some of those 1:1 comments public might be helpful for new devs interested in learning how these processes work
< sipa> *commemts
< jonasschnelli> IMO QRCode have redundant parts while - as far as I understand bech32 – has also error correction.. seems a bit unefficient. Though QRCode are another layer I guess.
< jonasschnelli> (need to read petertodd's response; wasn't aware(
< gmaxwell> jonasschnelli: He suggested it use more characters but using anything non-alphanum makes it so double click copy doesn't work in browsers, also being a non power-of-prime base makes strong error detection infesable, and at best could reduce the size by a couple percent.
< gmaxwell> These points are all addressed in the document, I believe.
< kanzure> tromp's MiXcAsE proposal was a joke right?
< sipa> gmaxwell: base 43 is actually totally doable as it is a prime
< petertodd> gmaxwell: there's a few things that the document doesn't mention as clearly as it could, like how non-alphanum poses barriers for non-english-speaking users
< wumpus> jonasschnelli: I don't think the qr error detection/correction is strong enough for somthing as critical to get right as an address
< gmaxwell> (in fact in an earlier revision we used - as a seperator but dropped it for click copy compatibility)
< sipa> gmaxwell: but it requires bignum code to convert
< jonasschnelli> wumpus: Yes. Agree.
< sipa> petertodd: feel free to suggest better language :)
< gmaxwell> wumpus: it would probably be fine if we only ever used QR codes.. but obviously that isn't possible.
< petertodd> sipa: will do - also thanks for having such a great list of rationals already! really helpful to understand why something was chosen and what the tradeoffs are
< gmaxwell> petertodd: yes, though really being num only be best for international support but the result is significantly bigger and slower to deal with. :( I tried that out at one point.
< wumpus> gmaxwell: a more compact format could be used with qr codes, in any case, if that's desirable. That doesn't affect this standard.
< luke-jr> QR codes could hypothetically encode it in binary, but meh
< kanzure> rationale section was good, although i think it would be worthwhile to describe the 'exhaustive search'
< kanzure> in public
< wumpus> luke-jr: right.
< gmaxwell> luke-jr: if we had a binary QR only format I assume it would intentionally begin with something like a null byte to make sure it doesn't get mangled by text channels.
< petertodd> gmaxwell: yeah, we're fortunate that lots of non-english-speakers still known the alphabet; even english speakers often don't know the names of punctuation symbols, and those names are also not entirely canonical
< jonasschnelli> It's probably not worth to design an additional binary addr standard for QrCodes. Maybe they are only temporary? Gone in 3-4 yrs?
< gmaxwell> There were a lot of other rational things we trimmed from the document. One of pieter's concerns was that the length of the document would make it seem complicated (though the proposal is pretty simple.); feedback on reddit seemed to somewhat validate that.
< petertodd> luke-jr: so by matchin QR code's special-case character set support, we end up with an encoding that at the low level is basically binary anyway
< gmaxwell> jonasschnelli: I think that it's probably not worth it for that, but industry feedback would be nice.
< gmaxwell> petertodd: thats a point, though the unfortuate thing is that it's exposed to mangling.
< sipa> the alphanuneric mode is pretty efficient; only 10% extra compared to binary
< sipa> it uses 5.5 bits per character
< gmaxwell> e.g. QR decoder -> something that munges up the strings -> software -> coins in sppppaccccceeee.
< sipa> so 5.5 bits per 5 bits of data is pretty goof
< sipa> good
< petertodd> jonasschnelli: I think the day Qr codes aren't heavily used will be the day bech32 is irrelevant anyway because people are all paying each other by very different means :)
< jonasschnelli> probably... :)
< sipa> while base58 uses 8 bits per 5.86 bits of data
< gmaxwell> sipas point is more important, only a 10%-ish overhead in this format vs some binary QR code.
< sipa> if you also drop the checksum, we get much larger savings of course
< sipa> in practice you'd get a 25% reduction for p2wpkh
< jonasschnelli> side note: could the Bech32 encoding also be used for private keys (== 32bit seeds)?
< gmaxwell> true. but ugh. the some text handling path replaces a character.
< sipa> jonasschnelli: good question!
< wumpus> indeed, for base58 addresses there also never has been a binary standard for qr codes
< sipa> jonasschnelli: we've been thinking about a stronger checksum for private keys
< Anduck> QR codes iirc can work even if 30% of the code misread
< wumpus> which would have saved a lot more
< petertodd> jonasschnelli: yes - in fact I'm planning on using it for non-bitcoin applications too
< sipa> possibly 12 checksum characters (which is the maximum doable with 64 bit arithmetic)
< jonasschnelli> Yes. An alternative for bip39?
< gmaxwell> For private keys additional error _correction_ is interesting, and additional overhead is not very consequential.
< sipa> bech32 can only correct 2 errors, but detect 4
< sipa> for addresses you really only care about detection
< sipa> but for private keys you want correction
< wumpus> yes, quite a different use case
< gmaxwell> So we've been working on that some in the background. So it would be a different spec but just very similar. (e.g. similar kind of 5 lines of code) for the 12 digit checksum or what not.
< sipa> with a 12 character checksum, correcting 3 errors is trivial
< jonasschnelli> current WIF can correct 0.
< sipa> but perhaps we can find one that can correct 4
< gmaxwell> kanzure: we left out a lot of technical manutia about the construction which is interesting but not really relevant to the spec.
< wumpus> makes sense to keep the spec focused and minimal, that way people will more likely read it :) you could link to an additional description, or include it as appendix or such
< sipa> earlier version explained finite field arithmetic and generator polynomials :)
< petertodd> wumpus: maybe good to make it a spec + design document? put rational/etc. in the latter. IIRC SHA3 had a document like that, among others.
< wumpus> petertodd: yes indeed
< jonasschnelli> The question about the level of complexity: We should't care. Pieter's stuff tend to be complicated. But only because its great and though through well. Awesome work IMO!
< jtimon> I haven't fully reviewed it, but I've had a fast read on several versions of the doc and asked questions to sipa
< gmaxwell> Finding a 12-digit code that corrects 4 may take more computing power than we have with just the a hundred cores here... though sipa has done a lot to take that search from completely intractable to plausable. :) I think this work is a lot lower priority though other things we could be working on (like utxo database refactors and tx compaction) ....
< sipa> jonasschnelli: i think the result is actually surprisingly simple
< sipa> (but i'm probably biased)
< gmaxwell> In any case, review and comment please!
< luke-jr> I think it's more important to *detect* errors, than to correct them
< sipa> luke-jr: for addresses, absolutely
< luke-jr> 4 seems a bit low in that regard
< jonasschnelli> sipa: I think deeply understanding the specs takes a coupe of hours... and it's different to the current base58check. This may frighten off people... but that shouldn't be something that should affect the process of selecting the *next* addr encoding format IMO
< gmaxwell> luke-jr: for private keys you can generate the resulting wallet and check against the blockchain.. so correcting can be very useful.
< petertodd> luke-jr: what do you mean by "4" in the above?
< luke-jr> [19:12:59] <sipa> bech32 can only correct 2 errors, but detect 4
< gmaxwell> Well it is _guarenteed_ to detect 4 errors, more than 4 you have a 1:2^30 chance of detecting.
< sipa> luke-jr: well it has a 99.9999999068% chance to detect more errors
< luke-jr> if I carefully construct gibberish, I'd expect it to be rejected
< luke-jr> sipa: ah, okay
< petertodd> luke-jr: for this application, the threat is random mistakes, not malice - in the malice case someone could just replace the address anyway
< gmaxwell> This graph shows the probablity of an error going undetected (y) as a function of how likely the user is to make mistakes (x):
< sipa> jonasschnelli: https://github.com/sipa/bech32/blob/master/ref/python/segwit_addr.py <- full encoder/decoder implementation
< cfields> great work indeed. From what I remember from our discussions before, the work to derive the function was complex, but afterwards, implementing it is simple. Seems best-case to me :). Will review now that it's final.
< jonasschnelli> sipa: 121 lines... yes. thats indeed very simple.
< gmaxwell> As you can see, if the user's error rate is below 3.53 errors per address entered, this code has better protection than a 32bit hash (like base58 check uses). And because of case modulation, users are MUCH less likely to make mistakes with this format.
< gmaxwell> If the user is unlikely to make errors, then the effective protection of this scheme tends to infinity.
< gmaxwell> e.g. 0.1% chance of error per character and the probablity that an errored string goes undetected is 1:2^60.
< sipa> enough on this topic, i guess
< wumpus> other topics?
< gmaxwell> jonasschnelli suggested the der wallet thing.
< jonasschnelli> Yes. We still keep all priv keys DER encoded in BDB
< jonasschnelli> I think we should bundle the next wallet db update with remvoing DER
< gmaxwell> The compressed flag has no effect on our operation. We should always set it IMO... make the wallet sightly smaller. Its dumb that the data is stored at all.. it's not used. But since it's there, better to make it small.
< gmaxwell> if we make the format incompatible, sure.
< jonasschnelli> Reading DER (old versions) must be supported. But new wallets should store in plain.
< wumpus> #topic DER privkeys in wallet
< gmaxwell> Why are we even storing seralized private keys when BIP32 is in use?
< jonasschnelli> Yes.
< jonasschnelli> Importprivkey
< jonasschnelli> and old wallets
< jonasschnelli> But I think HD wallets should only have a single privkey (the HD seed)
< gmaxwell> okay, for that, sure. But I agree, fine to not use DER private keys for that.
< jonasschnelli> Also the seed key is also DER right now. :/
< gmaxwell> but otherwise there shouldn't be seralized private keys. Pubkeys yes. :)
< gmaxwell> wut
< jonasschnelli> Yes. Ideally it should be... and I think we are working in this direction...
< wumpus> didn't we have an issue related to this?
< gmaxwell> okay well, whatever, there is only one there. We could encode it as cat pictures for all I care.
< phantomcircuit> yeah it's saved using the same format as other private keys
< gmaxwell> wumpus: there is an issue open over the compressed flag, though its inconsequential.
< wumpus> gmaxwell: ok
< wumpus> I wasn't aware it was inconsequential
< gmaxwell> I think that to the extent that we continue to write those keys (e.g. for old wallets) that we should always use the compressed flag.
< * luke-jr> ponders if importprivkey should be rejected for HD wallets
< wumpus> did anyone reply that to the issue?
< jonasschnelli> Though,... as long as nobody reads BDB from the outside it shoudn't matter
< gmaxwell> wumpus: we decided compressed/uncompressed from the pubkey.
< gmaxwell> wumpus: the public key inside the der private key does nothing.
< wumpus> luke-jr: would be nice to keep HD wallets 'pure'
< wumpus> luke-jr: at the least it should throw a warning
< wumpus> gmaxwell: weird :)
< jonasschnelli> luke-jr: Yes. We should reject... in that case
< luke-jr> if we could do warnings, importprivkey should always throw a warning anyway :p
< gmaxwell> Background: The DER private key format includes the public key, along with all the ECC group parameters, and other overheads all packed in hundreds of bytes of ASN1 parsing hell.
< jonasschnelli> wumpus: Yes. It wired... better DER is gone sooner then later
< jonasschnelli> It would be another thing that will make future devs stumbel
< wumpus> agreed
< wumpus> so if there is a backwards-incompatible wallet change, that could be included, I guess
< gmaxwell> Turns out we were trying to tell the der private key to use compressed or not on the basis of the key using compressed or not... but the code was wrong. But it doesn't matter because we never did anything with the private key embedded public key.
< jonasschnelli> wumpus: Yes. The hd chain split...
< jonasschnelli> (which we hopefully can merge soon)
< wumpus> jonasschnelli: right
< jonasschnelli> I think we would have enought time post hd split merge to remove DER for 0.14
< gmaxwell> we should stage up several incompatible changes to make at once, the multitude of versions is a support burden. Changing the private key formats would be one of them.
< luke-jr> jonasschnelli: 0.14 is gone; you mean 0.15?
< BlueMatt> 15
< jonasschnelli> Ergh.. yes.
< jonasschnelli> Do we expect that the wallet db handles error correction or should we directly switch to something like Bech32 for our internal privkey serialisation?
< BlueMatt> gmaxwell: the code looks the same either way? wallet version 14990 14991 14992 can all be in the same release, whatever...code is still wallet.AmUpgradedTo(THING)
< gmaxwell> in our code but not necessarily in other tools.
< gmaxwell> also we have to worry about interactions.
< BlueMatt> fair
< wumpus> the wallet db has no error correction
< BlueMatt> gmaxwell: to be clear, are you suggesting they all happen in one wallet version, or in one release?
< BlueMatt> (I mean we dont support wallet versions from mid-release, but it'd be really annoying...)
< jonasschnelli> wumpus: I think this would/should be very desirable
< wumpus> (berkeleydb doesn't and neither does leveldb, the latter only has detection)
< wumpus> jonasschnelli: yes, encoding privkeys in a resilient format makes sense
< wumpus> jonasschnelli: though doesn't encryption break that?
< wumpus> jonasschnelli: the error correction should be around the encryption, not inside it
< wumpus> otherwise one bit change will completely mangle it anyway
< gmaxwell> BlueMatt: preferable to be one version, but certantly one release. I don't know that we really don't support midrelease wallets. "sorry dude, your funds are gone, you made the mistake of helping us test." :P there are degrees of non-support.
< luke-jr> we have never broken wallet versions from mid-release AFAIK
< jonasschnelli> wumpus: good point
< gmaxwell> bech32 is not helpful for the kinds of errors we would find on disk.. you'll usually lose a whole sector at a time.
< wumpus> eh, wallets from mid-release should not suddenly break in later updates
< jonasschnelli> There is also a PR i'd like to bundle with chain-split/rm DER that would detatch the client version from the wallet version/migration handling (PR is called "wallet flags" or similar)
< gmaxwell> To make a wallet database master key error robust you should just repeat it across multiple disk sectors. :P
< wumpus> gmaxwell: I've seen single bit errors happen on disk too, though usually due to bus errors
< gmaxwell> wumpus: yea, but not as common. And it would be fine to have a wallet format with a 4kb master key. .. but I think thats stuff to think about for the post BDB world.
< petertodd> wumpus: repeating across multiple disk errors fixes that too :)
< wumpus> but some of the corrupted block files I"ve analyzed certainly had single bit errors
< gmaxwell> sorry I stated it too strongly.
< wumpus> in any case - for HD wallets we need to store way less private keys
< wumpus> so blowing up the master key to 4kb is fine
< gmaxwell> ya, t'was my intended point. :)
< wumpus> if that helps protect against various kinds of corruption
< BlueMatt> gmaxwell: ok, so multiple version numbers, but external tools only need support them as a group and -walletupgrade only supports them as a group?
< sipa> the wallet flags idea removes that release/walletversion inconsistency completely
< gmaxwell> BlueMatt: I think that would be okay but why multiple versions? add the code with the features, then a patch that turns all of them on with a single version.
< sipa> gmaxwell: that's very hard for testing
< sipa> as you'd need to be able to override the min version in tests to be able to even observe the feature
< BlueMatt> yea, was just thinking easier for review burden, but I suppose thats fine too
< BlueMatt> sipa: not so hard to have an extra commit in a pr that doesnt get merged
< sipa> BlueMatt: ugh
< gmaxwell> we just create support hurdles for ourselves supporting mixed features. "oh this change brakes bob's wallet because he had a mid-release version that had X and not Y"
< sipa> code in master should be testable
< sipa> wallet flags does not necessarily imply supporting mixed features
< gmaxwell> As long as it doesn't then I don't mind.
< gmaxwell> Just please lets not dribble in three different incompatible versions during a release cycle.
< BlueMatt> jonasschnelli: detaching client version, fine, but would strongly prefer a serially increasing version number and not flags
< gmaxwell> and end up with people with weird wallets that will break later.
< sipa> we could also detach walletversion from client version
< sipa> that would accomplish the same thing
< BlueMatt> yea, that ^
< gmaxwell> unless we're going to add tests for all the possible variations we shouldn't let people running master encounter them unawares.
< sipa> and just increment the wallet version when a new feature is added
< BlueMatt> wasnt sipa magic number picker for a while? can we just go back to that
< BlueMatt> or jonasschnelli, since he owns wallet these days, mostly
< gmaxwell> for testing you should just be able to set a commandline paramter that turns on future versions/features.
< gmaxwell> e.g. default wallet created is version X but there is code for version Y (or flag Y) and you can ask for it for testing, but until its the default its unstable and all bets are off.
< jonasschnelli> Need to think about it. Flags seems to be good anyways (supports HD, watch only,...) but most not be linked to the wallet database version.. hmm..
< gmaxwell> I don't care if we retain compatiblity for a wallet someone created by running bitcoind with --force-wallet-screw-me-over-now-now-now
< jonasschnelli> s/most/must
< BlueMatt> jonasschnelli: flags just makes things more complicated to support (interactions, yuck), and I'm not sure we have a compelling use-case for turning on some stuff but not others, aside from "I want my wallet to work all the way back to version 0.X"
< gmaxwell> And I'm fine with flags too, of course, but not with supporting arbritary combinations. (unless someone really wants to build extensive automated testing for all the combinations)
< luke-jr> BlueMatt: Core isn't the only compatible wallet.
< gmaxwell> e.g. basically a version implies flags... and you use flags in the code, and that doesn't mean that all combinations of flags get tested.
< BlueMatt> luke-jr: all other wallets that read our wallet format are unsupported, I believe (and, hell, probably easier for them if we dont have flags with interactions...)
< jtimon> right, you can have flags and maybe even have debug option for experimental wallet features that we can remove in releases (or just not leave any wallet feature that doesn't have a version yet)
< jonasschnelli> Flags would allow flexible backporting? But I think we don't want to do this.
< jtimon> or just an extra commit that doesn't get merged like BlueMatt says
< luke-jr> BlueMatt: you're rather aggressive with "unsupported" IMO
< wumpus> we never backport wallet features
< gmaxwell> luke-jr: Our _binary_ wallet format isn't intended to be interoperable. If it is, great! but without a spec and a moving target, it isn't something we can guaretee. To begin with there would need to be an extensive all features all flags test suite... and I think it would be a waste of time to construct that for this format (esp since we're never going to specify BDB itself).
< BlueMatt> luke-jr: we have limited resources, but, fair point
< wumpus> yes, the database formats are not an external interface
< wumpus> we reserve the right to change them completely from one release to another without advance notice, though that will usually not happen
< gmaxwell> I don't know if wallet format compatiblity is a reasonable goal in general, at least not this year-- just because it's so integrally tied to the functionality.
< wumpus> older wallets should *always* work
< wumpus> that's the only guarantee
< gmaxwell> but obviously we don't want to create unnecessary incompatiblity.
< gmaxwell> right.
< gmaxwell> okay, is this horse dead yet?
< wumpus> yes
< jonasschnelli> yes
< wumpus> other topics?
< sipa> lunch?
< gmaxwell> I'd like to know if people might be willing to sign onto a statement like this:
< luke-jr> wallet format changes move slowly. I try to avoid backporting pending PRs to Knots now, but IMO once they get merged they should be fair.
< gribble> https://github.com/bitcoin/bitcoin/issues/4 | Export/Import wallet in a human readable, future-proof format · Issue #4 · bitcoin/bitcoin · GitHub
< jtimon> sorry, "--force-wallet-screw-me-over-now-now-now" is exactly what I meant
< petertodd> gmaxwell: ACK
< * BlueMatt> proposes we have a weekly call for "what pr are you blocked on and want review on", though we've had mixed results when I've done that before
< gmaxwell> (gee thanks for the link gribble)
< wumpus> #topic statement against running closed source node code
< kanzure> "If it ever does, you may safely assume that the actual contributors to the project are locked in a basement somewhere. In that case, please send help." and food
< luke-jr> gmaxwell: I haven't thought about it sufficiently to be certain about "never"
< wumpus> gmaxwell: ACK
< petertodd> gmaxwell: though I think the statement should be a little more clear on what the risks are - this is money so such a binary may very well be an attempt to steal your coins
< wumpus> or may have hidden vulnerabilities, bugdoors and backdoors
< BlueMatt> gmaxwell: seems reasonable
< jtimon> BlueMatt: the original flags can be replaced with the version flag once you have a version for a set of them
< kanzure> yep seems good to me
< gmaxwell> luke-jr: I think I have, plus 1001 other open source packages have. As I point out in the document even if all other options fail, "shut off" remains an option too.
< jonasschnelli> Yes. A statement would be good. Also.. github.binaries with a SHA256 on reddit is _not_ want you want for a such ecosystem
< luke-jr> true
< petertodd> wumpus: see, I think it's good to say in such a statement "hidden vulnerabilities, bugdoors and backdoors" and *also* give a 100% concrete example for the non-devs reading it
< jtimon> I mean, you can have a single fWalletForNextVersionExperimental flag
< gmaxwell> okay well I would welcome revisions. I didn't want to make it long and explaining all the options and risks.
< gmaxwell> Just saying "don't run these things if 'we' do" is protective to both the users and us.
< wumpus> we have this stringent gitian process which costs quite a lot of time, maybe we should put more focus on it...
< gmaxwell> In any earlier draft I had listed out some of the options we have (and have used before) and it was 3x longer than this just explining a couple of them. :)
< kanzure> also, submitting source code "later" is not acceptable either
< achow101> the statement should say "Bitcoin Core project", not "Bitcoin project" IMO
< wumpus> I sometimes have the feeling we're doing all this auditability for nothing if people are ready to just download random crap binaries of the internet and run them, though I guess some do validate it...
< gmaxwell> wumpus: well we know not everyone does that.
< sipa> wumpus: i think it's a slow process... the best we can do is talk about it, and highlight what we're doing
< sipa> and just the increase in number of gitian builders on itself is a good sign
< kanzure> mentioning bugdoors, backdoors and other malware is important
< sipa> (reminds me, i need to build 0.14...)
< petertodd> wumpus: I know for a fact that many high-value users do verify PGP signatures and the like, e.g. exchanges and custodians. That effort isn't wasted.
< gmaxwell> okay, well I can work with some people on some language twiddling. Sounds like a couple people are interested.
< wumpus> I guess the important thing is that some people do validate it, and can raise a stink if things don't match, which is not possible with random crap executables
< wumpus> petertodd: right.
< gmaxwell> and luke-jr if you want to talk later and have me convince you that we'd would never in the future do a binary release, I'd be happy to.
< BlueMatt> one of these days I'm gonna finally convince cfields to finish his "trusting trust" compiler-builder for use in gitian, then I'll be much louder...the trusting-ubuntu part of gitian is...irritating at best (but still infinitely better than the alternative)
< gmaxwell> (_NEVER_ I mean, in the strongest sense, I'm sure you already are convinced that almost never would hold)
< luke-jr> yeah, I certainly can't think of any circumstances that would ever justify it.
< cfields> BlueMatt: yea, that's a goal of mine for 0.15.
< petertodd> gmaxwell: I like that you explicitly mentioned in that doc that interruption of service is perferable
< BlueMatt> cfields: yippee
< BlueMatt> topic?
< * BlueMatt> proposes we have a weekly call for "what pr are you blocked on and want review on", though we've had mixed results when I've done that before
< gmaxwell> petertodd: well I tested some of this explination on reddit while talking to people 1:1 about some things. so I have an idea of what needs to be explained.
< jtimon> gmaxwell: I would prefer if we mentioned free software explicitly or even replace open source with free software, but I agree with everything said in that statement
< sipa> BlueMatt: sounds good to me
< sipa> BlueMatt: everyone gets to list 1 PR?
< BlueMatt> sipa: sure
< jonasschnelli> Maybe two? :)
< BlueMatt> mine is 9725
< gmaxwell> jtimon: ack. Thanks for the feedback I'll revise some and consult with others that had comments.
< jonasschnelli> #9725
< gribble> https://github.com/bitcoin/bitcoin/issues/9725 | CValidationInterface Cleanups by TheBlueMatt · Pull Request #9725 · bitcoin/bitcoin · GitHub
< wumpus> so jonasschnelli's wallet chain split seems important
< jonasschnelli> yes. please. #9294
< gribble> https://github.com/bitcoin/bitcoin/issues/9294 | Use internal HD chain for change outputs (hd split) by jonasschnelli · Pull Request #9294 · bitcoin/bitcoin · GitHub
< kanzure> on that last topic, uh, maybe wee should point out that backdoors have been found in node software already (ahem some exchanges come to mind..).
< BlueMatt> list from last week was #8694, 9294, 9725 and #7729
< gribble> https://github.com/bitcoin/bitcoin/issues/8694 | Basic multiwallet support by luke-jr · Pull Request #8694 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/7729 | An error has occurred and has been logged. Please contact this bot's administrator for more information.
< BlueMatt> any additions/replacements or should we let the list stand?
< gmaxwell> #9959 could probably take some more reviews.
< wumpus> I'd also really like to have feedback on my patches to add UNIX sockets support at some point (#9979, #9919) though it's not really urgent
< gribble> https://github.com/bitcoin/bitcoin/issues/9959 | Mining: Prevent slowdown in CreateNewBlock on large mempools by sdaftuar · Pull Request #9959 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/9979 | p2p: Bare minimum to support UNIX sockets by laanwj · Pull Request #9979 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/9919 | UNIX sockets support for RPC by laanwj · Pull Request #9919 · bitcoin/bitcoin · GitHub
< gmaxwell> (suhas doesn't appear to be here now, so I thought I'd mention it)
< * jtimon> reminds to build 0.14
< luke-jr> I still have to look over BlueMatt's review on multiwallet, so I'll refrain from listing one this time ;)
< gmaxwell> wumpus: Those are a great idea. I will be trying them out. Did we find out if your patch to libevent is going to make it upstream?
< cfields> wumpus: ah, i hadn't seen the p2p one
< jonasschnelli> Can we keep the most important 1PR list on github somehow? A project?
< sipa> jonasschnelli: sounds good to me
< jonasschnelli> I think this would be great because priorizing reviews is hard right now
< wumpus> gmaxwell: still haven't heard about that one, but I don't think it should matter for getting these patches in
< sipa> a PR can belong to multiple projects, right?
< gmaxwell> keep in mind to prioritize things that we would want in a 0.14.1.
< wumpus> sipa: AFAIK yes
< jonasschnelli> hopefully
< luke-jr> PR labels seems workable for that too
< wumpus> we already have so many labels
< luke-jr> "Top Priority" or somethign perhaps
< luke-jr> wumpus: but labels are easy to filter :D
< wumpus> also I use those to classify pulls for the release notes, so I'd prefer not to use them for non-categoritical things
< jonasschnelli> Project seems to be the better choice...
< wumpus> yes
< gmaxwell> "author:wumpus" (since he does most of the merging any of his PRs are effectively 1-reviewer disadvantaged for getting in)
< sipa> DONG
< wumpus> #endmeeting
< lightningbot> Meeting ended Thu Mar 23 20:00:29 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< gmaxwell> Okay, I think its time to feed sipa.
< luke-jr> lol
< jonasschnelli> hehe
< jtimon> I've been waiting on #8855 (previously #6907 previously part of other prs) for so long...
< sipa> must... not... chew PRs
< gribble> https://github.com/bitcoin/bitcoin/issues/8855 | Use a proper factory for creating chainparams by jtimon · Pull Request #8855 · bitcoin/bitcoin · GitHub
< kanzure> programmer chow is other programmers
< gribble> https://github.com/bitcoin/bitcoin/issues/6907 | Chainparams: Use a regular factory for creating chainparams by jtimon · Pull Request #6907 · bitcoin/bitcoin · GitHub
< * jonasschnelli> will be in SF in September.
< luke-jr> when did we agree on a 1 hour limit to meetings anyway? IIRC there was only a set start time.. maybe they should go for 24 hours? :p
< sipa> jonasschnelli: cool!
< gmaxwell> luke-jr: because without a limit some people would starve.
< sipa> jonasschnelli: i may be in switzerland next month
< kanzure> luke-jr: maybe it should be based on time until 6 blocks
< jonasschnelli> sipa: If you are,... tell me and let's hand out.
< luke-jr> gmaxwell: am I the only one who eats during meetings sometimes?
< luke-jr> kanzure: hehe
< jtimon> another old one with several previous attempts is #8498 but I guess that's my fault for saying "Optimization" without doing benchmarks...
< jonasschnelli> s/hand/hang
< gribble> https://github.com/bitcoin/bitcoin/issues/8498 | Optimization: Minimize the number of times it is checked that no money... by jtimon · Pull Request #8498 · bitcoin/bitcoin · GitHub
< BlueMatt> we always have office space available for those who want a stopover between .eu and sf :P
< BlueMatt> and we have good flights :)
< luke-jr> does SF<->EU go that direction?
< jonasschnelli> BlueMatt: Ah.. man. I fly to SF over Iceland!
< jtimon> I suspect #9608 will also be open for ages, but at least that will force me to read every change in ProcessMessage...
< gribble> https://github.com/bitcoin/bitcoin/issues/9608 | Net: Divide ProcessMessage in smaller functions by jtimon · Pull Request #9608 · bitcoin/bitcoin · GitHub
< BlueMatt> jtimon: we said you get to pick one :P
< sipa> jonasschnelli: cool, wowair?
< cfields> jtimon: I'll chime in on that one.
< jtimon> just like #8329 and #8337 force me to review every change in certain functions
< jonasschnelli> sipa: Yes. :|
< gribble> https://github.com/bitcoin/bitcoin/issues/8329 | Consensus: MOVEONLY: Move functions for tx verification by jtimon · Pull Request #8329 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/8337 | Consensus: MOVEONLY: Move functions for header verification by jtimon · Pull Request #8337 · bitcoin/bitcoin · GitHub
< jtimon> BlueMatt: if only one, then #8855, actually it doesn't need rebase but I'll just do it in case there's new uses of Params(std::string)
< gribble> https://github.com/bitcoin/bitcoin/issues/8855 | Use a proper factory for creating chainparams by jtimon · Pull Request #8855 · bitcoin/bitcoin · GitHub
< jtimon> cfields: cool thanks, rebased, no new uses of Params(std::string)
< jtimon> gmaxwell: is there anything planned for 0.14.1 ?
< sipa> bug fixes
< bitcoin-git> [bitcoin] jtimon closed pull request #9845: RPC: cleanups in rpc/server (master...0.15-extern-rpc-server) https://github.com/bitcoin/bitcoin/pull/9845
< BlueMatt> wumpus: were we gonna label peoples' review-picks as "blocking someone" or should i just put up a list somewhere
< gmaxwell> jtimon: there are a number of fixes already merged.
< jtimon> I see
< gmaxwell> jtimon: I'd like to see one ship in the not far future. I'm kinda surprised how few bugs we've had with 0.14.0
< BlueMatt> I believe the list was me: 9725
< BlueMatt> jonas: 9294
< BlueMatt> wlad: 7729 (+9979, 9919, though not blocking per se)
< BlueMatt> suhas: 9959
< BlueMatt> jorge: 8855
< jtimon> I think it had to do with delaying forking the 0.14 branch, or perhaps I just want to think that
< jtimon> BlueMatt: thanks, that's cool, perhaps we can do that at the end of every meeting
< BlueMatt> jtimon: that was my plan
< jtimon> well ack your plan then
< BlueMatt> though I think jonasschnelli has things to respond to from my review on 9294
< jtimon> that will save me "review begging" comments
< BlueMatt> oh, and I'll go ahead and add alex as 9942, because iirc he has a few things based on that too
< sipa> segwit nodes don't download blocks from non-segwit nodes, by design
< gmaxwell> *when segwit is active.
< bitcoin-git> [bitcoin] tjps opened pull request #10067: [trivial] Dead code removal (master...tjps_dead_code) https://github.com/bitcoin/bitcoin/pull/10067