< fanquake> promag I can run them also
< warren> MarcoFalke: I have had python-vm-builder RPM packages for years but it broke sometime this year and couldn't figure out how to fix it. although I use only qemu-kvm, not lxc.
< bitcoin-git> [bitcoin] Varunram opened pull request #11793: Docs: Bump OS X version to 10.13 (master...patch-1) https://github.com/bitcoin/bitcoin/pull/11793
< Varunram> ^ This needn't be a standalone PR in itself, can be merged with other fixes
< bitcoin-git> [bitcoin] laanwj closed pull request #9254: [depends] ZeroMQ 4.2.2 (master...zeromq-4-2-0) https://github.com/bitcoin/bitcoin/pull/9254
< bitcoin-git> [bitcoin] laanwj closed pull request #11793: Docs: Bump OS X version to 10.13 (master...patch-1) https://github.com/bitcoin/bitcoin/pull/11793
< aj> jnewbery: apparently github detects merge conflicts even when git merge and rebase both work fine, so i've split the preliminary changes from #11774 into #11796; hopefully 11796 can be easily merged once acked, since it shouldn't break any pending PRs
< gribble> https://github.com/bitcoin/bitcoin/issues/11774 | [WIP] [tests] Rename functional tests by ajtowns · Pull Request #11774 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/11796 | [tests] Functional test naming convention by ajtowns · Pull Request #11796 · bitcoin/bitcoin · GitHub
< wumpus> github is really sensitive with regard to merges, more than the stock git commands
< wumpus> might be so that at least no one blames them for merges going wrong
< aj> wumpus: yeah, it does make sense. bit weird not being able to tell quite what github thinks the conflict is though
< wumpus> at a company I used to work for the integration dept had a similar rule. Two changes affect the same file? let the devs sort it out
< wumpus> yeah it could use some better reporting
< wumpus> something I'd really like on github would be a graph of which open PRs collide with which other PRs
< wumpus> would help a lot in finding the optimal merge order, as well as notify authors early if their PR is going to need to be rebased
< sipa> that would be very useful
< sipa> n^2 complexity though in the number of PRs, i think
< wumpus> worst case, though the n^2 could be filtered by excluding PRs that affect completely distinct files
< wumpus> also one'd want to avoid recomputing the whole thing, just incrementally take opens/closes into account
< wumpus> (ugh, and PRs that are re-pushed)
< aj> that just makes it O(n) for each PR change though
< wumpus> which is reasonable, you can do it in the background, not at the time someone opens the graph
< aj> yeah, n isn't /that/ large
< wumpus> though yeah if github wants to do it for all projects it could get bad for them
< wumpus> I don't really care if it would be a paid-only feature or such
< wumpus> I've changed "Docs and output" label to just "Docs", it is only for documentation, for logging changes and such please use "Utils/logging/libs" instead.
< wumpus> I think lumping everything output-related in one issue was a bit confusing
< wumpus> in one label*
< aj> wumpus: had a quick play, and it seems like you can get somewhere with detecting PR conflicts using git locally. for xa in $tsts; do a=pull/origin/$xa; if (git reset --hard origin/master && git merge $a -m "merge $a to master") >/dev/null; then for xb in $tsts; do if [ "$xa" -ge "$xb" ]; then continue; fi; b=pull/origin/$xb; (git format-patch $(git merge-base $a $b)..$b --stdout | git apply --check
< aj> -) > APPLY_${xa}_${xb} 2>&1 && echo "no conflict between $xa $xb" || echo "merge conflict $xa $xb ?"; done; else echo "conflict with $a and master"; fi; done > CONFLICT
< aj> gah
< aj> wumpus: had a quick play, and it seems like you can get somewhere with detecting PR conflicts using git locally. -- was what i was trying to say
< aj> wumpus: https://gist.github.com/ajtowns/d0cf97678dc83efdf3f6cbf7083a35a0 -- was what i was trying to paste
< wumpus> awesome!
< wumpus> pretty clever that you manage to avoid touching the working tree
< wumpus> but yes seems I won't get around to setting up a git tree that pulls all the PRs locally
< aj> wumpus: hmm? pulling all the PRs locally is easy and awesome though?
< wumpus> yeah no problem, just don't do it at the moment to avoid cluttering my tree further, I have 858 branches of my own at this point :)
< fanquake> git branch -D *
< aj> wumpus: the pull requests don't show up in git branch -av for me, so i don't even notice (actually, i don't even know how to get a list of them...)
< wumpus> fanquake: git branch -D doesn't take a pattern :-) you'd need something like git branch | xargs git branch -D
< wumpus> aj: interesting, I'd have expected them to show up
< wumpus> (I regularly do 'git branch --list 'pull/*' | xargs git branch -D' to get rid of the temporary branches left behind by github-merge.py in some cases)
< aj> ah, "git remote show origin" will at least give me a list of them
< wumpus> apart from that I use a single repository with worktrees for persistent branches like 0.15
< wumpus> anyhow it works pretty well, I think it will still work pretty will with 231 more branches; will have to find something to clean up closed PRs though
< aj> updated that gist with the compatible and conflicts graphs for a handful of recent PRs
< wumpus> or does that automatically remove removed branches at the gh side as well?
< wumpus> aj: oh wow
< aj> wumpus: the only thing i can think of to make it useful is dumping data into python to work out which collections of PRs have complete subgraphs of compatability
< wumpus> so a connection means a conflict? why do 11790 and 11741 conflict?
< wumpus> 11741 only changes two files, yet conflicts with 9 PRs or so
< wumpus> it's possible :)
< aj> yeah, they merge fine manually; git apply --check complains about multiwallet.py, rpc/rawtransaction.cpp and test/txvalidation_tests.cpp
< aj> ah, i'm trying to apply patches from master twice
< wumpus> but 11741 touches neither of those
< wumpus> hehe okay
< fanquake> nice
< wumpus> oope, so fetch = +refs/pull/*/head:refs/pull/origin/* doesn't just fetch *open* pull requests
< wumpus> whee tracking 8288 branches now
< wumpus> happy I did it in a temporary repo :)
< wumpus> aj: great
< Varunram> aj: thanks for the gist :)
< aj> probably needs to work out which PR branched off from master most recently and use that as a base
< wordsToLiveBy> I'm reading the bitcoin white paper, and in section 11 Satoshi gives the calculations for how you can prove with probability that an attacker can't outpace the combined processing power of the rest of the nodes, but I am not quite getting the math being used.
< andytoshi> #bitcoin please, this is just about software development
< jnewbery> aj: thanks. I've reviewed #11796. It should be an easy review/merge for others.
< gribble> https://github.com/bitcoin/bitcoin/issues/11796 | [tests] Functional test naming convention by ajtowns · Pull Request #11796 · bitcoin/bitcoin · GitHub
< aj> jnewbery: ugh, what sort of person points out a bug and says not to fix it /o\
< saint_> any idea why can't bitcoin core download the blockchain on a remote NAS drive ?
< wumpus> saint_: ask in #bitcoin please, will answer there
< saint_> okay
< jnewbery> aj: I wanted to get in before someone else pointed it out and told you to fix it :)
< jnewbery> I don't think it matters since the next PR should remove that code anyway
< achow101> is the github git notifier dead?
< wumpus> achow101: intermittently, yes
< wumpus> achow101: it seems incredibly unreliable
< jonasschnelli> I wonder how the Twitter commit feed works... I expect them polling
< wumpus> I think it must be; it isn't a webhook that we set up
< wumpus> we only have a slack webhook, and the IRC integration
< wumpus> so having ruled out that it listens on IRC, it must either be listening on slack or polling
< wumpus> not sure whether the slack notiifications do go through
< wumpus> if the normal webhook works we could set up our own IRC bot
< jonasschnelli> wumpus: slack works...
< wumpus> jonasschnelli: great, so we isolated it to a problem with their IRC bot, not their notifications in general
< instagibbs> meeting in 3?
< jonasschnelli> Looks like
< instagibbs> or am i off
< wumpus> you're right
< wumpus> if the bot stays flaky like this tomorrow I'll contact github support, it's already been for a few days
< wumpus> #startmeeting
< lightningbot> Meeting started Thu Nov 30 19:00:03 2017 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
< wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag
< cfields> hi
< sipa> hi
< CodeShark> hello
< achow101> hi
< Provoostenator> hi
< instagibbs> hi
< jonasschnelli> hi
< wumpus> #topic high priority for review
< meshcollider> hi
< luke-jr> hi
< wumpus> 8 PRs have been tagged for https://github.com/bitcoin/bitcoin/projects/8
< BlueMatt> why does cfields have 2?
< wumpus> I added cfields's BanMan last week as it's blocking his further net refactoring
< cfields> BlueMatt: wasn't me, but i'll take it :)
< wumpus> ohh is it unfair?
< * jonasschnelli> also have two :|
< instagibbs> achow101, i see you rebased yours? we shooting for post-0.16?
< BlueMatt> i think we try to have only one per person
< jtimon> hi
< BlueMatt> well i suggested we include jonasschnelli's other one as its kinda a segwit-wallet-blocker imo
< BlueMatt> or at least in-same-release
< achow101> instagibbs: I'm shooting for 0.16
< achow101> but it may not make it
< instagibbs> ok, I can give it more love, though I think I've given a lot already, probably needs others....
< achow101> still some segwit related things to do and runn1ng wants simulations
< achow101> also been busy with exams and classwork
< wumpus> well, good solution for that, review and merge one of cfields's PRs and there will be only one left :)
< karelb> achow101: I came just in time :) yes I wanted some simulations
< jonasschnelli> wumpus: heh
< sipa> so i think we're maybe unintentionally moving from a "we release regularly with whatever is finished" to "X is a blocker for release Y"
< BlueMatt> oh, wait, i dont have one...hmmmm, I'd say #10279
< wumpus> sipa: segwit is the only exception to that
< cfields> i have no problem with removing one of mine if it's an issue. But yes, I think 11363 is ready/easy to review
< gribble> https://github.com/bitcoin/bitcoin/issues/10279 | Add a CChainState class to validation.cpp to take another step towards clarifying internal interfaces by TheBlueMatt · Pull Request #10279 · bitcoin/bitcoin · GitHub
< BlueMatt> sipa: I think we'll go back to that post-segwit-wallet, no?
< wumpus> sipa: for the rest, releases are only ever blocked on bugfixes not features
< achow101> sipa: I think 0.16 is the exception? because segwit wallet
< wumpus> yes
< jtimon> sipa: what is the blocker for 0.16 ?
< sipa> okay, that's fine - i'm not saying it's a bad thing
< instagibbs> i think it's openly acknowledged
< wumpus> and segwit wallet is not intended as a *blocker* for 0.16
< wumpus> we intend to release 0.16 early after that's in
< sipa> fair enough
< wumpus> so it's more like a trigger
< sipa> but it still changes prioritization a bit
< BlueMatt> indeed
< instagibbs> i think the segwit pr is shaping up nicely at least
< wumpus> if segwit wallet takes longer than the 0.16 release window, then IMO we should release 0.16 without it
< wumpus> but I don't expect that
< BlueMatt> wumpus: agreed
< jtimon> oh, I see, we want to release 0.16 soon after segwit wallet gets in
< wumpus> jtimon: yep
< sipa> i'm about to push some review fixes to #11403
< gribble> https://github.com/bitcoin/bitcoin/issues/11403 | SegWit wallet support by sipa · Pull Request #11403 · bitcoin/bitcoin · GitHub
< wumpus> jtimon: seems more advisable than trying to backport the whole lot + dependencies to 0.15
< cfields> sipa: not sure if you've clarified somewhere, do you consider all of the listed TODOs in 11403 blockers for release as well?
< jtimon> wumpus: yeah, I think I suggested that, eithe way it makes sense to me
< wumpus> jtimon: thanks in that case :)
< luke-jr> branch 0.15 into 0.16 ;)
< sipa> cfields: yes
< cfields> ok, thanks
< gmaxwell> So, segwit-wallet is done?
< jtimon> I'm not following review on the segwit wallet pr, how is it going?
< sipa> i think #11403 is pretty much done - just nits in command line option handling and output
< gribble> https://github.com/bitcoin/bitcoin/issues/11403 | SegWit wallet support by sipa · Pull Request #11403 · bitcoin/bitcoin · GitHub
< jtimon> great
< achow101> I tried reviewing it but it's big and scary
< wumpus> awesome
< sipa> we may want to discuss what to do with things like how dumpprivkey and signmessage should work in a post-segwit world
< * Randolf> congratulates achow101 for trying
< cfields> i found it reasonable review on a per-commit basis
< instagibbs> cfields, same
< sipa> as i believe some projects have come up with formats on their own
< cfields> *to review
< achow101> sipa: trezor has implemented their own segwit message signing thing
< jtimon> I guess I need tofix the nits in #8994 and #10757 if I want them to have a chance to get merged before 0.16 then
< gribble> https://github.com/bitcoin/bitcoin/issues/8994 | Testchains: Introduce custom chain whose constructor... by jtimon · Pull Request #8994 · bitcoin/bitcoin · GitHub
< achow101> I think that might be the only one though
< gribble> https://github.com/bitcoin/bitcoin/issues/10757 | RPC: Introduce getblockstats to plot things by jtimon · Pull Request #10757 · bitcoin/bitcoin · GitHub
< sipa> achow101: perhaps we can just adopt that
< jtimon> what are we referring to by "soon after"?
< wumpus> sipa: well if other project's import/export formats make sense, it'd be good to use them for interchangeability
< achow101> sipa: I don't think it's documented anywhere though
< meshcollider> Should there be a BIP to formalise it
< achow101> at least not yet
< Provoostenator> Is there a good standard for message signing?
< sipa> longer term i think a script-based signing would nice, but i don't think we're ready to adopt something like that
< gmaxwell> I thought they did but backed it out because it wasn't standardized yet. What they did didn't seem to awful, though I feel a little uneasy about the mutability between keytypes.
< karelb> sipa: ad sign message - in trezor we changed v constant - https://github.com/bitcoin/bitcoin/issues/10542#issuecomment-316032523 - no BIP, no time :(
< sipa> karelb: ok, i'll have a look
< sipa> the original bitcoin core message signing doesn't have a bip either
< luke-jr> Provoostenator: no
< wumpus> a BIP is not a requirement for us implementing it, though in parallel it'd be nice
< BlueMatt> sipa: it probably should if we change it, though
< achow101> sipa: perhaps it should? retcon one :p
< sipa> seems reasonable
< wumpus> good to already have implementations and the BIP just formalizes it
< gmaxwell> Well, signmessage is generally not very good; but fixing it should be a longer term thing.
< Provoostenator> I vaguely remember trying to implement / refactor signing and being not amused by the lack of a standard.
< luke-jr> current signed messages get very little real use, and most "usage" is based on misconceptions
< karelb> sipa yes good point. I wanted to write up a document that describes both current and segwit message signing, but I got stuck on how ecdsa signing actually works :)
< wumpus> luke-jr: agree
< luke-jr> IMO it'd make sense to just deprecate it until a better replacement is made
< wumpus> luke-jr: it's not a priority at least
< instagibbs> luke-jr, it's used for airdrops :P
< sipa> luke-jr: i agree it's not all that useful
< Provoostenator> Can it be made coin-agnostic as well?
< wumpus> certainly shouldn't be a blocker for 0.16
< luke-jr> instagibbs: that's a misuse, since it doesn't prove you have funds ;)
< gmaxwell> Provoostenator: no.
< karelb> we didn't have segwit message signing in web wallet, but users repeatedly demanded it
< instagibbs> luke-jr, but i profit from it, how can it be misuse ;P
< instagibbs> +1 tho
< gmaxwell> The fact that it's inherently incompatible with multisig is a real bummer.
< luke-jr> karelb: for what, though?
< achow101> instagibbs: lol
< gmaxwell> luke-jr: airdrops
< luke-jr> XD
< wumpus> using bitcoin keys for anything else than signing transactions continues to make me uneasy
< jtimon> yeah, +1 to move to signing messages based on scripts
< luke-jr> MAST-based signmessage would be a sensible replacement
< Provoostenator> jtimon: what would that look like?
< wumpus> do hardware wallets even support it?
< luke-jr> Provoostenator: you could have a MAST if-branch that is always false for transactions, and then true for meta uses
< gmaxwell> wumpus: If I were to it again, I'd make signmessage work by just creating a transaction which is inherently unminable. It would make it a lot more compatible, though somewhat larger.
< CodeShark> gmaxwell: a general solution for airdrops seems extremely difficult
< jtimon> Provoostenator: like we sign txs but signing any message, in elements we sign blocks so perhaps you want to take a look (since there's generic functions to sign stuff, but they're not exposed and only used for blocks)
< gmaxwell> In elements sipa wrote a patch for a script based signmessage; but we ran into ... challenges.. with how softforks would be handled.
< wumpus> gmaxwell: that'd have been better; at least the keys would only be used to sign things in one format
< gmaxwell> Segwit script versioning would fix those challenges.
< CodeShark> yes, two chains might have very different redemption conditions for the same utxo
< Provoostenator> Hah, so you can do multisig messages? :-)
< sipa> Provoostenator: yes
< sipa> Provoostenator: and timelocked signatures :p
< Provoostenator> Or even partial messages?
< jtimon> gmaxwell: I think they should be handled with flags
< sipa> jtimon: yes, but the general property of softforks doesn't apply
< jtimon> oh, right, script versioning solves it too
< sipa> you don't want a "oh, i don't know this signature version! it's valid!!"
< gmaxwell> jtimon: the 'pubkey' needs to commit to the rules. pre-segwit style softforks don't.
< gmaxwell> Well you can tristate: Valid, invalid, unknown public key version.
< wumpus> right
< luke-jr> sipa: that's a risk for txs too, though
< jtimon> sipa: sure, I mean, this is out of the blockchain, so you just need to know which flags to use when validating...what gmaxwell said, commit to the rules
< gmaxwell> prior to having the explicit versions we couldn't do that, which is why we never even merged the patch in elements, much less brought it to bitcoin.
< luke-jr> why sighash flags can't fail valid
< gmaxwell> luke-jr: attacker controls signature side flags.
< luke-jr> gmaxwell: right, that's my point
< gmaxwell> same reason sighash flags can fail valid in bitcoin: If I pick an out of range sighash flag I could forge signatures for your pubkeys.
< sipa> ?
< CodeShark> I also don't follow
< luke-jr> [19:16:56] <sipa> you don't want a "oh, i don't know this signature version! it's valid‼" <-- this is a problem for transactions just as much as for signed messages
< wumpus> are there any topics we really want to discuss in this meeting? I think we're drifting off a bit
< gmaxwell> You cannot trigger "unknown behavior, I'll let it pass" based on sighash flags like you can based on the content of public keys.
< cfields> next (quick) topic suggestion: codesigning keys update
< jnewbery> Other PRs I'd love to see merged for v0.16 are #10583 #10579 #11415 . They're all removing wallet dependencies from server, but are API changes so have a one release deprecation lag time before the dependency can actually be removed.
< gribble> https://github.com/bitcoin/bitcoin/issues/10583 | [RPC] Split part of validateaddress into getaddressinfo by achow101 · Pull Request #10583 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/10579 | [RPC] Split signrawtransaction into wallet and non-wallet RPC command by achow101 · Pull Request #10579 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/11415 | [RPC] Disallow using addresses in createmultisig by achow101 · Pull Request #11415 · bitcoin/bitcoin · GitHub
< jtimon> yeah, how about what "shortly after" means for 0.16 ?
< jonasschnelli> Two topic proposals: 1) review "nits" reduction, 2) bitcoin core code signing assoc.
< jonasschnelli> 2) goes in hand with cfields topicp.
< wumpus> jnewbery: thanks, I think adding all of them to high priority is a bit much, but we could add one
< wumpus> #topic codesigning
< cfields> I just wanted to point out that after researching for a bit, there's no painless way to renew our osx cert (without involving the btcf), so I think it's prudent that we explore other options.
< * cfields> passes the mic to jonasschnelli
< achow101> wumpus: it seems like some of those are ready to be merged. they have utACKs and no comments left
< jonasschnelli> The association stands and apple has accepted the entity
< jonasschnelli> Code sining certificates are ready (need to setup the key)
< cfields> jonasschnelli: oh that's fantastic! nice work :)
< wumpus> achow101: if they're ready for merge even better, please notify me of those things outside the meeting
< jnewbery> wumpus: thanks. I think they're all pretty much ready for merge. achow101 has been doing a great job maintaining and rebasing them - perhaps just a bit more ACKing and they can go in
< jonasschnelli> We need a windows code signing cert... have never done that but I'm ready to order if someone gives instructions where and how
< wumpus> PSA: if something is ready for merge just let me know here instead of waiting for me to stumble on it accidentally :)
< BlueMatt> jnewbery: afaict only maybe the last is ready? the second only has one ack?
< achow101> wumpus: ok!
< BlueMatt> jonasschnelli: that was fast!
< wumpus> jonasschnelli: oh great!
< cfields> achow101: note that a driver cert is different. More requirements.
< wumpus> getting a driver cert is much more difficult
< achow101> oh, didn't see that that was for drivers
< wumpus> I guess it should be, with the privilege level it operates at
< jonasschnelli> The question is if we want to try that distributed RSA key
< wumpus> we do
< BlueMatt> gmaxwell: mic?
< BlueMatt> or who was it who said they were gonna look into hooking it into a reasonable way to use it?
< cfields> I'll work on the CSR handling as promised. I assume we'll just have to shove the result back into the expected format.
< gmaxwell> I didn't think we were going to bother to do it for the first one. but this is going faster than expected! :)
< jonasschnelli> cfields: Yeah. Apple needs that -----BEGIN CERTIFICATE REQUEST-----
< achow101> BlueMatt: that was gmaxwell. I also wanted to take a look at it. it looked painful
< gmaxwell> it's not so painful but we need a process for converting raw RSA numbers into csrs and what not, which cfields was going to look into.
< gmaxwell> The easiest thing to do may be to just do trusted setup: generate the key normally on an offline machine, and we can split it after the the fact.
< BlueMatt> ugh
< gmaxwell> the MPC signing is radically simple and faster than MPC key generation.
< BlueMatt> i thought the mpc generation was implemented?
< gmaxwell> Though the stuff I linked to does both. (though it's setup for only three parties atm)
< BlueMatt> i mean thats probably fine?
< BlueMatt> 3/3, I assume?
< achow101> gmaxwell: can it be done faster than what is already implemented?
< gmaxwell> achow101: of course, but do we care if it takes hours?
< wumpus> if it's a one-time thing we don't
< cfields> jonasschnelli: let's talk after the meeting. It'd be great if we could get a dummy to test with.
< gmaxwell> BlueMatt: yes 3/3. Generation you'd always do as n/n then potentially reshare to a different threshold.
< jonasschnelli> cfields: sure
< BlueMatt> its not like this is holding a zksnark trusted-setup for all btc
< gmaxwell> How big are the keys they use for these things? 2kbit or 4kbit?
< Provoostenator> BlueMatt: would be nice to get another blog from P
< Provoostenator> l
< sipa> ?
< BlueMatt> ?
< Provoostenator> Peter todd riding down Canada to generate his signing key.
< jonasschnelli> gmaxwell: 2048 is what my apple RSA keys are
< Provoostenator> And then bragging that's more secure than Zcash
< wumpus> indeed, it's just code signing, for one OS, if something is signed that is not validated by the gitian process that's discovered soon enough anyway
< gmaxwell> Provoostenator: it's totally unrelated.
< jonasschnelli> Code signing won't give a lot of additional security...
< jonasschnelli> It's just a UX thing IMO
< gmaxwell> zcash has a backdoor for the whole system, this is just some code signing crud.
< wumpus> right
< Provoostenator> I know.
< gmaxwell> the users shouldn't care AT ALL about the MPC we use for it, the purpose of the MPC is to protect the developers personally from being implicated in the unfortunate case their own computers get hacked.
< sipa> does the MPC generation have a trusted setup?
< gmaxwell> sipa: no.
< sipa> i assume not - if you're fine with trusted setup, just generate a single key and split it layer
< sipa> ok, so it is entirely unrelated
< * jtimon> is curious about the "review nits reduction" topic...
< wumpus> yes it's just to prevent the person doing the code signing from becoming a specific target
< jonasschnelli> review nit reduction topic?
< gmaxwell> sipa: it just uses paillier, and lots of roundtrips. it's pretty simple.
< wumpus> #topic review nit reduction
< jonasschnelli> I just feel that we spend a lot of time in finding code-style nits, fixing code-style nits and some of the PRs are almost a single wall of nits... It doesn't seem to be efficient... I think..
< jonasschnelli> We should enforce the rules... ( I know I already brought that up )
< jonasschnelli> The libcurl project does that... it won't compile (make) if the code style doesn't matches
< jonasschnelli> It would reduce the review time a lot...
< jonasschnelli> And we apperently fixing the nits anyways at some point in time
< jtimon> you mean getting travis to complain about style nits?
< Provoostenator> Currently we're only running whitespace linter?
< jonasschnelli> jtimon: travis already does this... (whitespace)
< BlueMatt> if there were an easy way to apply it to new code only, I'd say thats probably fine
< jonasschnelli> I just want to get the code styling nits away from github comments
< * jtimon> nods
< BlueMatt> but it was my impression we were not able to find a way to do so
< jonasschnelli> BlueMatt: only new code. Yes.
< jtimon> BlueMatt: that was my understanding too
< wumpus> so have e.g. clang-format check the diff of pull requests?
< sipa> jonasschnelli: (brainstorming) or we could have a bot that marks a PR ready for review only after all nits are fixed
< jonasschnelli> wumpus: Yes. Can we enforce that somehow?
< jonasschnelli> sipa: Would also be something.. yes.
< sipa> so that people know not to look at something while there is still automated review going on
< wumpus> jonasschnelli: I don't know...
< Provoostenator> It would also help to offer some hints for developers how to run linters in their editor. I'll add an entry for OSX Atom once I figure it out myself.
< jtimon> didn't someone wrote something like that at some point?
< wumpus> sipa: as long as it doesn't generate noise (posts) that's ok with me
< wumpus> sipa: so if it works like travis, just adds another cross to the status
< jonasschnelli> Would something speak against enforcing the clang-format-diff check during make?
< wumpus> w/ a link to the list of problems
< sipa> wumpus: right,t hat would be nice
< meshcollider> BlueMatt: adding new linters to Travis would only affect PRs now since #11699
< gribble> https://github.com/bitcoin/bitcoin/issues/11699 | [travis-ci] Only run linters on Pull Requests by jnewbery · Pull Request #11699 · bitcoin/bitcoin · GitHub
< wumpus> but I don't want any bots that generate notifications in my mail
< sipa> right
< jonasschnelli> wumpus: Yes, That would disturb even more
< jonasschnelli> Just an indication if its ready to review...
< jonasschnelli> And so we can have less of those "brackets, spaces missing blabla"
< * BlueMatt> was always in favor, did we find out if we could get a reasonably stable output out of clang-format?
< BlueMatt> or was it always very version-dependant?
< jonasschnelli> It just feels some reviews are more or less a "clang-format diff output"
< wumpus> I like how golang handles that, they just have one true style
< wumpus> and their linter can be safely used in e.g. pull request checks, w/ no ambiguity
< jtimon> jonasschnelli: wouldn't we need to comply with clang format in the whole project before doing the clang check in make? has anyone seen haw many changes trying to apply clang-format to the whole project produces right now?
< morcos> do we have good developer documentation on how to do the right thing locally (not what the rules are, but how to check them?)
< sipa> you can apply clang-format on just the diffs
< meshcollider> jtimon: not if it's only run on the diff
< jonasschnelli> yes
< sipa> but clang-format only checks whitespace/newlines
< wumpus> morcos: good point!
< sipa> it doesn't check variable names etc
< jonasschnelli> morcos: we should focus on that
< gmaxwell> A small amount of code style nits on github are probably good for teamwork.
< wumpus> sipa: I think that's fine, those are the ones we want out of the way
< jonasschnelli> sipa: not sure if that is possible
< jtimon> meshcollider: right, only for pr diffs, yeah, I thought he meant every time you build locally
< wumpus> sipa: variable names are more of a human thing anyhow
< wumpus> sipa: so it's fine if humans comment on them
< morcos> i'd be happy to do more locally, i'm sure my PR's are disasters, but it would be nice if a good workflow was spelled out
< jonasschnelli> morcos: Indeed
< meshcollider> Scripted diffs will probably regularly require style change commits too to keep Travis happy
< jtimon> gmaxwell: there's always style nits that go beyond clang-format
< wumpus> I mean it could check basic things like is_this_snake_case for variable names in theory, but not whether it's a good name in the first place...
< sipa> sur
< jonasschnelli> yes
< jonasschnelli> It's more about the naming convenction (m_, snake_case, CONSTANTS, etc,)
< wumpus> and the latter is much more important for maintainablility :-)
< kanzure> do we have a clang-format file
< MarcoFalke> meshcollider: I'd prefer if we didn't use clang-format in scripted diffs. Makes it impossible to reproduce ...
< gmaxwell> jtimon: yes true enough, I guess my point was that they're not all bad.
< wumpus> kanzure: yes
< jonasschnelli> kanzure: yes
< wumpus> contrib/devtools/clang-format-diff.py
< wumpus> src/.clang-format
< kanzure> ah thanks.
< cfields> i assume it'd be possible to setup a git alias that shows the current diff, diff'd against the clang-format result
< * cfields> plays around
< jonasschnelli> MarcoFalke: are you up to write a higher level documentation for the clang-format-diff.py workflow?
< * MarcoFalke> proposed to add '.style.yapf' and hides
< jtimon> wumpus: right, that's what I remember being written
< MarcoFalke> jonasschnelli: I did
< MarcoFalke> git diff | cfd
< wumpus> yapf?
< gmaxwell> general problem with clang format is that it isn't stable across versions, or at least hasn't been historically. In general we don't have a highly consistent enviroment htat people contribute from.
< ryanofsky> fwiw, i am not bothered by nits whatsoever. at the very least they mean somebody is actually looking at my pr. i also use clang-format and clang-format diff all the time
< wumpus> gmaxwell: right, that has always bothered about it
< jonasschnelli> gmaxwell: yeah. that
< MarcoFalke> also what gmaxwell said
< MarcoFalke> wumpus: The same thing for python
< jtimon> well, the version can be fixed on travis, though
< wumpus> MarcoFalke: that's not very shocking
< jonasschnelli> ryanofsky: nits should still remain... just not stuff that computers can find and are covered by our code styling rules
< sipa> jtimon: right, but that makes travis effectively part of your workflow, which is unfortunate
< wumpus> you need to be able to run any checks that travis does locally
< sipa> jtimon: as in, you won't really know a PR is acceptable before pushing and waiting for travis
< gmaxwell> We could give developers a VM image or equivilent, but it's asking a lot.
< jtimon> sipa: right, or the concrete version if I want to run it locally first
< luke-jr> Travis is slow, too
< sipa> maybe someone should investigate how stable clang-format/ clang-tidy/ iwyu/ ... are
< morcos> my issue with the nits on PR's is it leads to sloppier review. When nits get fixed and potentially rebased. Prior reviewers can get sloppy about assuming that the final version is well reviewed, especially if its happened repeatedly
< jonasschnelli> Could we not do a check with clang-format-diff.py during "make" and at least place a bold warning (or even refuse to compile *duck*)
< achow101> jonasschnelli: it would blow up on existing code
< sipa> jonasschnelli: the problem is that different clang-format versions say different things
< jonasschnelli> sipa: significant?
< sipa> jonasschnelli: irrelevant, i think
< wumpus> jonasschnelli: 'make lint' would be nice
< sipa> and making something work on your own system, and then seeing travis complain about the exact same things but requiring something else would be pretty demotivating
< wumpus> jonasschnelli: that would just run all the checks, for C/C++, for python, for whitespace in docs, etc
< cfields> wumpus: +1
< gmaxwell> morcos: that is more a problem with the intermixing of nits and serious review, and also how github handles tracking comments (them magically vanishing when the code changes)
< jonasschnelli> wumpus: Maybe it should by bypassable, but the novice contributor should get warned if he uses invalid code-styling
< sipa> gmaxwell: that's no longer true with the 'review' function
< gmaxwell> Obviously we should just pull clang into our code tree and ship it too. :P
< sipa> you can see all former review comments, and the code they apply on
< wumpus> jonasschnelli: well, travis and the person that merge can also check
< jtimon> sipa: oh, that's what is for...
< instagibbs> sipa, interesting, more reason to use it
< wumpus> jonasschnelli: it just should be possible to do it locally too
< morcos> gmaxwell: yes, i'd argue that once serious review stars, nits should not be squashed, but should be left as a cleanup commit at the end. (at least of some kinds)
< sipa> jtimon: yes, you should use it :)
< wumpus> jonasschnelli: I'm not so much concerned with forcing people to run it but making it easy
< jonasschnelli> wumpus: yes. Travis code style check is not what we want in the first place (it helps,.. but you want to catch it before)
< sipa> something i have noticed is that sometimes 'nit' reviews start on PRs which are far from certain they'll be even concept ACKed
< wumpus> jonasschnelli: well if you don't do it before, travis will stop you and you need to wait longer, simple as that :)
< instagibbs> morcos, we might need to have guidelines for "ACK lockin"
< jonasschnelli> Okay... I'll have a look at lint and something simple within the make-process
< jonasschnelli> sipa: that also
< cfields> morcos: i have no problem with squashing nits as the pre/post-squash can be diff'd locally. It's rebasing to master that makes re-review a challenge imo.
< wumpus> sipa: yeah... then it adds even more nosie
< instagibbs> cfields, can be diffed if you locally checked
< sipa> cfields: right, i try to avoid rebasing, but i pretty liberally squash nits
< instagibbs> many times im just reading off of github(lazy)
< sipa> cfields: and i don't mind others doing that too, except for very complicated PRs
< instagibbs> unless it's a particularly intense series of commits
< achow101> instagibbs: that kind of breaks when there's lots of merge conflicts
< wumpus> sometimes I wish 'git fetch' could fetch by commit id to fetch individual commits, unfortunately that doesn't work
< cfields> pretty sure github can show the diff from a squash a well, you just have to come up with the url for it
< gmaxwell> well we have contributors who don't feel comfortable (or just don't have the expirence) to do much other than nit reviews. Their contributions are usually pretty helpful and I wouldn't want to ask most of them to stop. We could perhaps ask them to wait a day on new PRs so that there is at least time for Concept NAKs to show up first.
< wumpus> it can only be used with branch names
< wumpus> github can give you the patch for an arbitrary commit id though, but it's kind of annoying to do automatically
< gmaxwell> it is a little sad for someone to show up with a change, and then have two cycles on trivial changes before someone more expirenced comes in and says no-way-because-x.
< wumpus> gmaxwell: yeah...
< aj> gmaxwell: might be helpful to tag PRs as looking for concept acks vs looking for nits and style vs looking for tested acks and merging?
< MarcoFalke> wumpus: We could set up a bot to create branches for each utACK that is posted to gh
< instagibbs> not nitting a PR that doens't have any ACKs of any kind seems like a decent rule
< MarcoFalke> on a separate repo that is
< wumpus> MarcoFalke: that'd be kind of nice
< jtimon> cfields: at the same time rebasing is often good, for example, #8994 could unexpectedly start failing tests even if no new conflicts appear and github says it's all ready to merge it
< gribble> https://github.com/bitcoin/bitcoin/issues/8994 | Testchains: Introduce custom chain whose constructor... by jtimon · Pull Request #8994 · bitcoin/bitcoin · GitHub
< wumpus> rebasing is often necessary
< jonasschnelli> Yes. What aj says. The things is just, unenforced user rules tend to get ignored. :)
< Provoostenator> gmaxwell: as long as you learn something from the nits, it's not the end of the world having them concept-nacked later, imo
< MarcoFalke> then `git fetch bitcoin_reviewed_commits` will get you all the reviews
< gmaxwell> sometimes you don't know what you're going to do when you read it.. there are certantly PRs where I read them and then come away with no real opinions on it other than "you missed some braces here and there"
< wumpus> some files are hotspots and generate a lot of merge conflicts
< wumpus> although it's better now that main.cpp is dead
< gmaxwell> Provoostenator: no, but some people find it demotivating; 'they made me do more work then threw it out'. :)
< Provoostenator> True
< wumpus> Provoostenator: these are not the kind of nits that help you learn btter programming :)
< gmaxwell> Provoostenator: that can be resolved with a better perspective; bitcoin development is a distributed process, your effort is your contribution not the fact that it was merged, etc.
< jonasschnelli> gmaxwell: indeed
< gmaxwell> Provoostenator: but we can't really send every new contributor to a zen mastery class before they contribute.
< Provoostenator> Well, we could... but that would be flagged as spam :-)
< wumpus> Provoostenator: if it's a nit like 'it's better to use build in function X' or 'the loop can be more efficiently written like this' then I think it's great
< cfields> a slightly different work-flow might be: create a "fixed nits" commit on top of the branch, and squash all nits into that as they arise. Then merge that in without squashing.
< cfields> it makes for ugly commits getting merged in, but avoids the re-review after squash-for-nits issue.
< Provoostenator> So far I find most of the nits I received useful. They either taught me to look up coding practices, or motivated me to figure out how to run a linter.
< gmaxwell> a challenge with that is that we do get new contributions from people with zero git expirence.
< sipa> who think they need to open a new PR to fix a nit :)
< wumpus> the git basics stuff is explained pretty well in the contributing doc nowadays
< aj> do people think the +1, +0, -1, concept nak/ack thing from https://github.com/bitcoin/bitcoin/pull/11426#issuecomment-334091207 is good btw?
< wumpus> I've not gotten the question on how to squash a commit for a long time now
< gmaxwell> wumpus: I've noticed that, I wondered why.
< Provoostenator> Some projects make a giant squashed merge commit and just point to the original PR (e.g. AngularJS), but that's not ideal for other reasons.
< instagibbs> oh that's where -0 came from
< BlueMatt> aj: yes, I'm a fan
< wumpus> Provoostenator: we only want to encourage squashing when it's iterative changes, not atomic separate changes
< jtimon> aj: I would have preffered that BlueMatt had maintained a nack but answered my questions/rebute, to be honest
< BlueMatt> specifically caues we end up with lots of need for concept review
< BlueMatt> we have lots of prs where lots of folks are +0/-0 and dont think its worth review
< BlueMatt> but they sit open for months
< BlueMatt> which is bad
< MarcoFalke> +0 on end meeting
< BlueMatt> +1
< instagibbs> -0
< wumpus> oh, it's that time already
< wumpus> #endmeeting
< lightningbot> Meeting ended Thu Nov 30 20:00:33 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< instagibbs> so hold on, the numbers are for what? pre-ACK reflection on idea?
< BlueMatt> generally, ive been usint them as that, yea
< MarcoFalke> warren: I tested it in august. Should still work, as the vm-builder is compiled from source
< BlueMatt> see aj's link
< wumpus> did we just spend an hour talking about nits and code signing, oh - also the signmessage thing :-)
< BlueMatt> yes :(
< jtimon> BlueMatt: if they are worth merging I don't think it's bad they sit open for months, even if a couple of folks said -0
< instagibbs> wumpus, 0.16 release too
< BlueMatt> spent an hour talking about how we waste time on nits
< MarcoFalke> warren: We do that on debian and ubuntu, so I sticked with it
< wumpus> BlueMatt: lol!
< wumpus> instagibbs: yes, you're right
< morcos> didn't you guys have a libevent topic
< instagibbs> BlueMatt, more irc meetings until throughput increases
< BlueMatt> wumpus: <BlueMatt> oh, wait, i dont have one...hmmmm, I'd say #10279
< gribble> https://github.com/bitcoin/bitcoin/issues/10279 | Add a CChainState class to validation.cpp to take another step towards clarifying internal interfaces by TheBlueMatt · Pull Request #10279 · bitcoin/bitcoin · GitHub
< jtimon> so, yeah, how long is "shortly after" for 0.16 ?
< Provoostenator> Is there a safe way to make Travis cache more stuff? It's insanely slow.
< BlueMatt> fuck, we forgot to talk about cfields' libevent question :(
< BlueMatt> that actually mattered
< instagibbs> feel free to talk on it
< morcos> well if you, wumpus and cfields are here, you can still talk
< cfields> it's ok, I'm still investigating something locally
< gmaxwell> too bad we're not allowed to talk about things except in meetings :(
< Dizzle> Has libevent merged pass-in-your-own-socket yet?
< cfields> heh
< cfields> The specific issue is how to deal with #11785
< gribble> https://github.com/bitcoin/bitcoin/issues/11785 | Prevent file-descriptor exhaustion from RPC layer by vii · Pull Request #11785 · bitcoin/bitcoin · GitHub
< wumpus> Dizzle: I don't think so
< wumpus> Dizzle: at least not for the http client, if you mean that, there's ton of ways to pass your own socket but not there
< cfields> I'm not convinced that we can fix it entirely, so it remains unclear what to do about it
< Dizzle> wumpus: thanks, that's what I was wondering about. stratum wallet and stratum mining projects are looking forward to unix socket RPC
< gmaxwell> well we could always merge some code that tries to increase our FD count, but thats not a fix.
< instagibbs> aj I still don't see the exact relationship between the numbers and ACKs.
< wumpus> BlueMatt: you mean #10279 for high priority for review?
< gribble> https://github.com/bitcoin/bitcoin/issues/10279 | Add a CChainState class to validation.cpp to take another step towards clarifying internal interfaces by TheBlueMatt · Pull Request #10279 · bitcoin/bitcoin · GitHub
< BlueMatt> yes
< instagibbs> seems to be concept ACK on sliding scale
< wumpus> Dizzle: do you need it to be in bitcoin-cli for that?
< cfields> gmaxwell: the second part of the PR does that, but that makes me really nervous :(
< aj> instagibbs: +1 -1 = concept ack / concept nak; in between is "leaning this way or not, but not decided either way"
< wumpus> Dizzle: we could just merge the server side, then you can use it from your own code, just not the command line
< Dizzle> wumpus: nope, most pools and electrum servers use json-rpc themselves for that.
< Dizzle> that would be neat!
< instagibbs> aj ok so it is a replacement for concept acks... that makes more sense
< BlueMatt> mostly I love -0 - I dont think this is worth anyone's time to review, but if others want to, fine
< instagibbs> wumpus, speaking of merge ready, 10677
< Dizzle> BTW, if anyone's curious how Electrum implements message signatures on native segwit addrs, we just iterate through the possible address types until there's match: https://github.com/spesmilo/electrum/blob/66cce115ef93c913d65ef5c7d781c8065f79bbaf/lib/bitcoin.py#L632
< gmaxwell> cfields: don't we have to eliminate the use of select before we go increasing the default maximum beyond FD_SETSIZE?
< wumpus> Dizzle: I don't think it's awfully useful in bitcoin-cli anyhow though it's useful for testing (though OTOH, the patch also changed the RPC testing framework to be able to use UNIX sockets)
< wumpus> instagibbs: ok
< cfields> gmaxwell: I believe the limit becomes FD_SETSIZE*2 if you're using 2 select loops?
< wumpus> cfields: no
< gmaxwell> to my great shame I seem to have forgotten how select is implemented internally. :P
< wumpus> cfields: select limits the max fd, using multiple selects doesn't change that
< wumpus> we should probably have switched to poll a long time ago
< wumpus> IIRC there have been PRs for that, but rejected because libevent P2P was just around the corner...
< gmaxwell> cfields: we've been waiting forever for the great networking refactors to eliminate the use of select, but perhaps this is a mistake-- select on windows scales fine, and switchin to poll on *nix is a couple line patch IIRC.
< cfields> well regardless, libevent is using epoll/kqueue/etc. here, so the select limits don't (or shouldn't) apply to rpc
< wumpus> cfields: no, but the fd's used might be in select's limit!
< wumpus> cfields: if 1024 fd's are used, then select is dead, no matter who claimed them
< gmaxwell> cfields: yes, but if we increase the process FD count, we're going to end up with FDs with high numbers which I thought broke select but as mentioned since I seem to have forgotten how it works internally I could be confused.
< wumpus> gmaxwell: you're right
< wumpus> cfields: you're right that RPC itself won't be affected, but it still messes up P2P nevertheless
< cfields> ok, i was confused about the limit then.
< gmaxwell> cfields: basically as-i-fake-recall the FD number itself is converted into a position in the array, so if you only have 8 FD's monitored but one of them has number 21318 you're going to be in trouble.
< wumpus> gmaxwell: yup the fd number is converted to a bit position in the array
< wumpus> which is also what makes it so inefficient
< wumpus> if it is a sparse array....
< gmaxwell> I believe bluematt wrote a very small patch to change bitcoin to poll. We could take, that and wrap it in ifdefs so we still select on windows, and call that sub-issue done.
< BlueMatt> none of these things solve the problem....
< BlueMatt> the issue ends up being in eg leveldb
< gmaxwell> BlueMatt: no but it lets you increase the process limit above 1024.
< BlueMatt> i mean it'll blow up our p2p first, but mostly thats not a big deal
< wumpus> does leveldb use select?
< gmaxwell> since when does leveldb use select?
< wumpus> I don't hink so
< BlueMatt> wumpus: no, in this case we literally ran out of process fds
< wumpus> leveldb only croaks if you run out of all process fds
< wumpus> which is what vii's patch more or less solves
< gmaxwell> As I said above: increasing the count isn't a _fix_; ... but it's a pretty good mitigation.
< wumpus> at least the obvious 'attack RPC with tons of separate connections' doesn't work as an exhaustion attack anymore after that
< BlueMatt> increasing the count also mostly doesnt break p2p, p2p will just keep opening new sockets until it gets a low numbered one
< cfields> wumpus: unfortunately, it doesn't :(
< wumpus> cfields: oh?
< BlueMatt> its smart enough to handle it
< gmaxwell> BlueMatt: really? what? lol.
< cfields> wumpus: nah, i can still exhaust the FDs with little effort.
< BlueMatt> i mean its not *that* smart, eg it will fail each connection
< BlueMatt> but it wil lkeep trying to make connections
< wumpus> cfields: so we really need a fix at the libevent side?
< gmaxwell> BlueMatt: so if an incoming connection has a high FD number it just drops it?
< BlueMatt> yes
< wumpus> it's interesting as I did load tests when I started using libevent and never stumbled on this
< gmaxwell> thats really ugly, and doesn't let us increase the maximum, consider, we make the maximum 65536... and the most of your connections start getting dropped. :)
< cfields> wumpus: yes. Problem is that it does while(something){accept()...}
< wumpus> if I only had known its http server was so crappy :(
< cfields> so if you manage to make a shitload of connections while it's in that loop, it'll keep swallowing 'em
< BlueMatt> gmaxwell: i mean if your rpc is getting flooded you probably want to drop most connections :p
< BlueMatt> cause you're already overloaded
< wumpus> BlueMatt: yes, you want to drop them
< wumpus> BlueMatt: that would be the right solution, not hoard file descriptors
< gmaxwell> BlueMatt: yea sure but I think the FD assignment is next highest in range until it hits the maximum, not lowest available.
< wumpus> AFAIK fd assignment is lowest available on many OSes
< BlueMatt> hmm, my node doesnt seem to break and i run with high fd limit
< gmaxwell> okay, I needed to be wrong about at least one thing.
< BlueMatt> so....dont think so on linux?
< gmaxwell> BlueMatt: without the poll patch?
< BlueMatt> yes, without
< wumpus> try to close stdout and the next fd you open :-)
< BlueMatt> i stopped using it and you can still get something like 800 maxonncections
< gmaxwell> gessh comeone we should just switch to poll, if its *nix only it should be a dozen line patch.
< gmaxwell> wumpus: lol.
< * BlueMatt> 's point is that it is almost entirely irrelevant to this issue
< wumpus> it's still relevant though
< gmaxwell> well increasing the fd count would change the urgency.
< * BlueMatt> 's point is we can increase fd count today
< BlueMatt> without poll
< wumpus> maybe not for this specific issue, but there's no good reason at all we're still using select, it only has drawbacks
< BlueMatt> fair
< gmaxwell> I really don't want to troubleshoot mysterious connection failures from "your FD was too high"
< Dizzle> Does select do ok on macOS? If not, would need to add kqueue polling to those 12 lines of code.
< gmaxwell> oh right osx poll is broken isn't it?
< wumpus> how is poll broken on osx?
< gmaxwell> sounds like they've fixed it, broken it again, and fixed it.
< wumpus> lol I'm not surprised, even validating the root password seems to be too difficult for them
< cfields> iirc they're just flip-flopping on undefined return values?
< gmaxwell> yea, it seems like that the case discussed there should be easy to avoid.
< wumpus> they took poor old freebsd and turned it to crap
< Dizzle> Of note, in addition to the issue that blog mentions, macOS' poll doesn't work on devices.
< gmaxwell> but shiny plastic crap
< gmaxwell> we only need it on sockets.
< wumpus> indeed, that doesn't matter
< wumpus> windows can't do select on devices or files either,we don't use that
< gmaxwell> BlueMatt: where is that poll patch?
< BlueMatt> uhhhh, lost?
< BlueMatt> i dunno
< Dizzle> I feel like ifdefing kqueue isn't a big stretch if we're already ifdefing poll. Python project ended up with a selector selector API on top of all this: https://docs.python.org/3/library/selectors.html#selectors.DefaultSelector
< gmaxwell> Dizzle: no probably not but whats the gain?
< Dizzle> performance and it works on those few versions of OS X with the broken poll
< gmaxwell> the performance differences are negligible for our sorts of usage.
< cfields> Dizzle: there's an abstraction that's done-ish, waiting on pre-requisites to go in. It uses epoll/kqueue/poll/etc. The poll discussion here arose because presumably it'd be simple.
< wumpus> on the fd assignment, this prints "0" on linux 4.10.0 https://gist.github.com/laanwj/8125d4971728dcfe85f6f5bd09dd572f , so at least anecdotally linux seems to assign the first available fd
< wumpus> Dizzle: that's why the goal is to move P2P to libevent, it already handles all those polling methods
< Dizzle> cfields: this abstraction on libevent or bitcoin core?
< cfields> yea
< Dizzle> OK, great. Sorry for the noise then :)
< wumpus> we really don't want to replicate all of that
< gmaxwell> the general problem with more ifdef paths is less coverage, the majority of the developers are are on linux, as are the majority of serious technical users who will try strange things and actually report results.
< cfields> #11227
< gribble> https://github.com/bitcoin/bitcoin/issues/11227 | WIP: switch to libevent for node socket handling by theuni · Pull Request #11227 · bitcoin/bitcoin · GitHub
< gmaxwell> so we should have a strong general preference to minimize platform ifdefs just on the basis that if we introduce a bug in one of them, it may take a long time to discover and fix.
< cfields> wumpus: as a data point, even with #11785, i can exhaust all handles via rpc in a minute or two.
< gribble> https://github.com/bitcoin/bitcoin/issues/11785 | Prevent file-descriptor exhaustion from RPC layer by vii · Pull Request #11785 · bitcoin/bitcoin · GitHub
< wumpus> at least the low-level libevent stuff is well tested
< gmaxwell> yes, at least libevent gets testing by other people.
< cfields> So I'm not sure that it's worth adding work-around hacks
< wumpus> cfields: so what can we do? (except say "don't do this")
< wumpus> I mean it's kind of sad to DoS yourself
< bitcoin-git> [bitcoin] luke-jr opened pull request #11803: Bugfix: RPC/Wallet: Include HD key metadata in dumpwallet (master...bugfix_dumpwallet_hdkeypath) https://github.com/bitcoin/bitcoin/pull/11803
< wumpus> and outside of localhost I doubt you'll ever accomplish such a rate
< gmaxwell> Though I think our general expirence with dependencies (including, sadly, libevent) is that we still run into bugs in them... in part because everything everywhere is broken, and small brokenness in most things is just background noise. ... so the fact that dependency foo is widely used helps less than we might guess.
< wumpus> that's just a fact of life, everything has bugs
< cfields> wumpus: yea, i'm not really sure
< gmaxwell> split rpc into another process. :P
< wumpus> I mean we even find gcc bugs
< wumpus> if anything should be well-tested...
< gmaxwell> wumpus: yea, well I used to comment that I was worried about our testing because we weren't finding GCC bugs. Glad thats fixed now. :)
< cfields> gmaxwell: yea, i expected some opposition, especially as these bugs crop up. I'd be ok with doing our own abstractions if it came down to it.
< wumpus> cfields: could we patch it at the libevent side?
< wumpus> cfields: I think trying to fix it in bitcoind is looking at it the wrong way
< gmaxwell> I think we should fix libevent.
< wumpus> cfields: it's a libevent bug
< cfields> wumpus: yea, pretty easily I believe
< wumpus> cfields: trying to work around upstream issues (at least permanently) instead of fixing them is pretty not done in open source
< cfields> through it might kill performance in some applications, so I'm not sure they'd want it upstream
< gmaxwell> and in the short term apply some mitigations in bitcoin, like increasing the FD count (ugh but I really don't like counting on FD magnitude sniffing)
< cfields> *though
< gmaxwell> cfields: surely exausting a processes' FDs isn't a cool move.
< wumpus> cfields: I don't think they care about performance in the http server
< cfields> wumpus: well it's in the listener, which is intended to be generic. The http server is just a user
< cfields> but yes, I'll do up a patch
< Randolf> [Syntax]: That's a nice way to encourage people. I like that.
< gmaxwell> among other things, it can contribute to amazing security vulns and remote crashes when it makes it impossible to open /dev/urandom to get randomness.
< Randolf> Whoops, sorry, wrong channel.
< wumpus> cfields: so the bug is deeper than just the http server, and woudl affect other clients (such as tor) as well?
< wumpus> cfields: now I'm interested :)
< cfields> wumpus: I'm unsure who/what else uses the listener
< BlueMatt> cfields: having a limited queue is a simple patch upstream *should* do
< cfields> i assume their dns server, as a quick example
< BlueMatt> so you can crash their dns server with enough tcp traffic?
< cfields> sec, checking
< gmaxwell> FWIW speaking of urandom, we'll assert if the open fails, so if you didn't know it this exhaust issue probably also causes (safe) crashes for us.
< wumpus> gmaxwell: yes luckily we check for that
< cfields> ok no, they only use the listener internally for http
< BlueMatt> what else uses libevent's http server?
< wumpus> cfields: I have a tor checkout here, anything I can easily grep for to see if they use it?
< BlueMatt> anything should be crashable
< cfields> wumpus: evconnlistener
< wumpus> cfields: no matches
< cfields> whew :)
< wumpus> they also don't use evhttp
< wumpus> they do have a http client in src/or/directory.c but apparently they rolled their own there
< wumpus> eh not that the client would suffer from this
< wumpus> still, there might be tons of things out there that use it
< wumpus> so if this can be triggered over the network it might still be reasonably serious
< wumpus> gah why are code search engines so useless
< wumpus> find 1000 forks of libevent, of course they have that string
< cfields> testing a patch
< wumpus> cfields: cool
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/9e38d357447e...fbce66a98267
< bitcoin-git> bitcoin/master 680bc2c practicalswift: Use range-based for loops (C++11) when looping over map elements...
< bitcoin-git> bitcoin/master fbce66a MarcoFalke: Merge #10493: Use range-based for loops (C++11) when looping over map elements...
< Vision> where are the proxy settings stored for bitcoin-qt? I cleared the proxy settings in the Options dialog, and now entering the settings dialog causes a crash. definitely a bug.
< meshcollider> Vision: what operating system?
< Vision> meshcollider: windows
< meshcollider> Vision: in that case, the options will be in the registry, under HKEY_CURRENT_USER\Software\Bitcoin\Bitcoin-Qt iirc
< wumpus> Vision: what version?
< wumpus> (of bitcoin core)
< Vision> 15.1 64-bit
< wumpus> there was a bug in 0.15.0 which caused crashes in the settings dialog, this was fixed in and 0.15.1
< wumpus> ok
< meshcollider> Vision: please let us know what values are in the registry before you modify them
< Vision> will do
< Vision> cuz yeah this is kinda catastrophic
< wumpus> it's better to use the flag to clear the gui settings
< wumpus> it will automatically make a backup
< meshcollider> wumpus: depends if he wants to clear all settings or just fix the proxy issue manually though :)
< wumpus> run bitcoin-qt with -resetguisettings
< Vision> looks like addrProxy is :10
< wumpus> this resets the settings and creates a guisettings.ini.bak with the old settings for troubleshooting
< Vision> lemme change that alone and see if that fixes
< achow101> start with -resetguisettings and then drop the backup file here
< wumpus> how did you manage to get :10 in there?
< meshcollider> just :10 with no IP?
< Vision> wumpus: clearing the boxes.
< Vision> I think :10 was the port
< Vision> maybe I left the port in
< Vision> and cleared just the IP
< Vision> meshcollider: indeed
< achow101> oh, I know how it crashed
< wumpus> strange, it shouldn't save in that case, but yeah possible
< Vision> oh it saved alright.
< achow101> it splits on the colon and does [0] and [1] to get the right params. that'll be an index out of bounds
< achow101> if there's no IP and just :port
< Vision> aha :D sounds like a culprit achow101.
< Vision> yeah I guess I just cleared the IP and not port
< Vision> and fixing that in the registry DID clear up the cache.
< Vision> CRASH*
< Vision> wtf. I swear I'm not retarded.
< achow101> I don't know how it let that save though
< Vision> once I finish what I was doing I may try to replicate it
< Vision> I swear I saw the '10' jump into the IP box before the dialog closed.
< Vision> if that helps
< Vision> aaaand I think it crashed instantly at that point.
< wumpus> it's clearly possible, though it might be that you stumbled on a very unlikely race condition. The parsing code really shouldn't crash on invalid input anyhow.
< Vision> aye
< meshcollider> Vision: were you using port 10 before you tried to clear it or is that number completely random
< Vision> yep I was
< Vision> not random
< Vision> oh yeah well that's easy to replicate
< Vision> have 'connect through socks5 proxy' checked, have an ip and port in the box, clear IP and hit OK
< Vision> instant crash
< Vision> and :port gets saved to registry
< wumpus> the code here https://github.com/bitcoin/bitcoin/blob/master/src/qt/optionsmodel.cpp#L231 for ProxyIP and ProxyPORT needs to catch the condition where the list is empty, or not long enough
< meshcollider> Yep I have replicated the issue on 0.15.1
< meshcollider> wumpus: I believe that is the part where it is loaded from the registry, we should deal with this when it is saved I think
< wumpus> meshcollider: yes but the parsing should be fixed too, it'd be two lines of code or so
< wumpus> meshcollider: this is the so-manieth time we run into crashes there for one reason or another
< meshcollider> wumpus: yeah :/
< Vision> [0] days since last crash
< meshcollider> why would this validator return true for an empty string: https://github.com/bitcoin/bitcoin/blob/master/src/qt/optionsdialog.cpp#L337
< Vision> why is 9050 hardcoded in that
< meshcollider> it is the default port
< meshcollider> if no port is specified
< Vision> ah.
< Vision> just seems an odd place for it
< wumpus> that's tor's default
< Vision> (cuz it looks like it's already in src/qt/optionsmodel.cpp as default)
< wumpus> yeah, feel free to send a patch that makes it a constant instead
< titim> Hi
< titim> lapoda lapoda