< gmaxwell>
12:47 < paveljanik> ECDSA Key Extraction from Mobile Devices via Nonintrusive Physical Side Channels (http://eprint.iacr.org/2016/230.pdf)
< gmaxwell>
Indeed, we were aware that OpenSSL was vulnerable to this kind of attack (though the pratical exploitation of it is quite interesting) and stopped using it several years ago as a result.
< GitHub21>
[bitcoin] jtimon closed pull request #7310: MOVEONLY: Move consensus functions out of main (master...consensus-moveonly-0.13.99) https://github.com/bitcoin/bitcoin/pull/7310
< GitHub57>
[bitcoin] jtimon closed pull request #6907: Chainparams: Use a regular factory for creating chainparams (master...chainparams-factory-0.12.99) https://github.com/bitcoin/bitcoin/pull/6907
< GitHub133>
[bitcoin] jtimon closed pull request #7566: WIP: Implement BIP9 and get BIP113 to be ready to be deployed with it as an example (master...bip9-0.12.99) https://github.com/bitcoin/bitcoin/pull/7566
< GitHub153>
[bitcoin] makevoid opened pull request #7636: Add bitcoin address label to request payment QR code (master...request_payment_qrcode_address_label) https://github.com/bitcoin/bitcoin/pull/7636
< jouke>
bitcoind -help-debug checks if bitcoin core is running?
< jouke>
bitcoind -help does not. Intended behaviour?
< jouke>
oh, --help -help-debug
< jouke>
never mind
< GitHub198>
[bitcoin] jtimon closed pull request #7563: libconsensus-p2a: Decouple pow.o from chain.o and move it to the consensus package (master...libconsensus-p2a-chain-cpp-interface-0.12.99) https://github.com/bitcoin/bitcoin/pull/7563
< GitHub137>
bitcoin/master 5ecfa36 Jonas Schnelli: Remove openssl info from init/log and from Qt debug window
< GitHub137>
bitcoin/master 7f001bd Wladimir J. van der Laan: Merge #7605: Remove openssl info from init/log and from Qt debug window...
< GitHub45>
[bitcoin] laanwj closed pull request #7605: Remove openssl info from init/log and from Qt debug window (master...2016/02/rm_openssl_log) https://github.com/bitcoin/bitcoin/pull/7605
< GitHub59>
[bitcoin] laanwj closed pull request #7586: fixes/refactoring for building against LibreSSL (master...2016/02/fix_openssl_libressl) https://github.com/bitcoin/bitcoin/pull/7586
< GitHub6>
bitcoin/master 5a2b1c0 Alex Morcos: Don't resend wallet txs that aren't in our own mempool
< GitHub6>
bitcoin/master 3368895 Wladimir J. van der Laan: Merge #7521: Don't resend wallet txs that aren't in our own mempool...
< GitHub64>
[bitcoin] laanwj closed pull request #7521: Don't resend wallet txs that aren't in our own mempool (master...testBeforeRelay) https://github.com/bitcoin/bitcoin/pull/7521
< jouke>
wallet config has spendzeroconfchange=0, but with 0.12 it does create transactions with inputs that were not confirmed yet.
< wumpus>
jouke: ugh! can you open an issue?
< wumpus>
strange, I wonder what changed in that code, looking at CWalletTx::IsTrusted() it still returns false when depth is not >=1 and that option isn't set
< wumpus>
so is it spending outputs that aren't IsTrusted?
< wumpus>
shouldn't be, AvailableCoins is always called with fOnlyConfirmed, and it skips coins that aren't trusted if that is set
< wumpus>
no, I don't see how this could happen :/
< wumpus>
maybe there was a reorg, and a transaction went from 1 to 0 confirms?
< jonasschnelli>
jouke, wumpus: interesting... testing local
< jonasschnelli>
Can't reproduce in a local test.
< jonasschnelli>
-spendzeroconfchange=0 worked for me
< gmaxwell>
What things need to be discussed today?
< CodeShark>
Versionbits, segwit status
< gmaxwell>
okay lets start on versionbits for now and we'll see what else gets raised?
< gmaxwell>
#topic Versionbits (BIP9)
< btcdrak>
hi
< sipa>
i'm about to push a few changes to 7575 (non semantic ones), and it should be ready for review
< gmaxwell>
Sipa has been working on refining the proposal and has made some recent changes which I think are pretty good-- eliminate some corner cases around start/stop.
< btcdrak>
The BIP update is looking nice.
< CodeShark>
Yes, I like the latest changes
< sipa>
so BIP9 currently guarantees that as long as the start/end times of deployments are non-overlapping, the bits are never ambiguous
< sipa>
so no need for dependency tracking between different deployments, just choose start/end times sanely
< CodeShark>
Yes, that's what I had in mind in my implementation but sipa did it better :)
< sipa>
7575 currently implements that, and has tests (for the low-level logic, not for the integration with consensus logic)
< gmaxwell>
I continue to be a little concerned that the activation threshold may be too high considering the low variance triggering mechenism, and activation delay. But I see nothing to do about that except try it and see if our first versionbits fork attempt fails to activate in a reasonable time.
< sipa>
we can reduce the threshold if needed
< sipa>
increasing is harder, as it may cause warning to not fire
< sdaftuar>
sipa: is 7575 going to eventually include deployment code for BIP68/112/113, or are you going to remove the last commit for a different PR?
< sipa>
sdaftuar: going to remove the last commit, and replace with whatever is agreed
< gmaxwell>
Thats a good argument. (that it's easier to reduce the threshold)
< btcdrak>
sdaftuar: I have the deployment code done for VB
< morcos>
sipa: should the regtest window be smaller than 2016?
< sdaftuar>
btcdrak: ok great. i was just going to say that saving the deployment for a subsequent PR might be easier for reviewing tests, etc
< morcos>
just seems like it'll make the tests less cumbersome if you want to watch what happens as you go up and down through a couple different windows
< btcdrak>
I was going to say, regtest with 2016 retarget is cumbersome
< gmaxwell>
we need to fix regtest to not fall over at retargeting.
< sdaftuar>
i think that is fixed
< morcos>
didn't we do that
< gmaxwell>
oh sorry! :)
< sdaftuar>
but it still might be cumbersome to generate long chains
< sipa>
yes, regtest just never changes difficulty now
< btcdrak>
it's cumbersome to generate long chains, since there are two retarget windows required.
< sipa>
but good point; i can set the regtest window/threshold lower
< cfields>
whoops, present. thanks gmaxwell.
< btcdrak>
sipa: +1
< gmaxwell>
why is typing setgenerate 4032 a problem?
< sdaftuar>
however i also worry that we're no longer testing mainnet parameters, and the consensus parameters are duplicated for each chain
< sipa>
gmaxwell: you want generate 4032
< btcdrak>
gmaxwell: it's too much for RPC tests
< sipa>
gmaxwell: setgenerate starts the internal miner with the specified number of cores; it no longer has special casing for regtest
< morcos>
it just takes a little longer...
< gmaxwell>
I do like to avoid avoidable differences between regtest and mainnet.
< gmaxwell>
perhaps the answer if it's taking to long is to make regtest even faster?
< sipa>
reintroduce SSE mining code? :p
< btcdrak>
:p
< gmaxwell>
I meant lower the difficulty. :)
< morcos>
12 secs
< sipa>
the regtest difficulty is 1/2000000000
< sipa>
you can at most get a 2x speedup
< morcos>
i think it would make the rpc test for this pretty slow as i imagine you'd need to do that many times
< gmaxwell>
OK, suggestion withdrawn.
< * paveljanik>
is late, sorry
< sdaftuar>
i worry more that a typo in the mainnet chain's deployment bitmask might go unnoticed/untested
< gmaxwell>
(why is it so slow? 6 seconds for 4k blocks seems like a lot)
< sdaftuar>
would anything catch that?
< sipa>
i'm still fine with lower window for regtest
< gmaxwell>
sdaftuar: review; I guess. (hahaha)
< btcdrak>
gmaxwell: it's much slower on RPC tests
< sdaftuar>
especially if there's stuff in your mempool right?
< sdaftuar>
blockindex consistency checks and mempool consistency checks both add up i guess
< gmaxwell>
So, sipa what do you need now for versionbits?
< sipa>
let me push a few changes, and then review
< sipa>
and tests are welcome
< gmaxwell>
#action after sipa pushes a few changes; reivew/test 7575, review BIP9
< gmaxwell>
Move on to segwit status? anyone have other agenda items to add?
< paveljanik>
feefilter review etc. please
< morcos>
and i hae a quick comment on tx backlog
< paveljanik>
BIP113
< gmaxwell>
k, lets do txbacklog right now.
< Luke-Jr>
I still think feefilter should be a little more flexible.
< gmaxwell>
#topic txbacklog
< Luke-Jr>
is there one?
< morcos>
i was wondering what kind of improvements are acceptable for minor releases
< paveljanik>
s/113/133/
< sdaftuar>
CPFP mining! :)
< sipa>
morcos: in response to an urgent problem, i'd say "anything"
< morcos>
i've noticed block validation seems to have slowed down significantly.. my theory is this is due to the daily cache flush and now many txs in blocks are older than that
< morcos>
this isn't urgent for sure
< sipa>
ok
< gmaxwell>
Right now there has been an increase in tx with fees over 1 satoshi per byte. The months standing background spam load of a around a gigabyte below that seems largely unchanged to me.
< morcos>
but it seems to me if we can correctly fix the "write but don't flush" aspect of the coinsviewcache, then that should be a significant performance boost
< morcos>
i guess it depends on whether we think validation times are a significant bottleneck for anything
< sipa>
morcos: yes...
< gmaxwell>
morcos: I've noticed the startup checks being much slower and was wondering if we'd made some regression someplace. Haven't tried bisecting.
< petertodd>
morcos: until we change to sending blocks before validating them they do add up
< Luke-Jr>
has anyone looked into whether the new txs are real or spam?
< gmaxwell>
Luke-Jr: some people have, petertodd was tweeting some analysis that strongly supported the latter.
< petertodd>
Luke-Jr: yeah, they look like long chains where eventually everything goes back to the sender, apaprently
< petertodd>
Luke-Jr: but no formal writeups exist yet
< Luke-Jr>
hmm
< morcos>
heh, you mean short chains.. woo hoo for chain limits!
< Luke-Jr>
any patterns to identify it with?
< petertodd>
morcos: no, they're long chains - once the txs confirm the chain is extended further
< gmaxwell>
in general most wallets are responding well (hurray! big improvement from 6 months ago) though not all.
< petertodd>
gmaxwell: speaking of, I noticed greenaddress has rbf code in their github repo
< morcos>
it looks to me like the backlog is diminishing as well
< petertodd>
gmaxwell: (for sending)
< gmaxwell>
petertodd: interesting, yes.. gait has been working on that; I think he was off in a design rathole on how to best support updating with additional outputs.
< petertodd>
gmaxwell: yeah, lots of possible ways wallets can do that, some of them quite different from how wallets work right now
< gmaxwell>
FWIW, with the new proposal for schnorr aggregate signatures, updating for more outputs will be much more attractive.
< cfields>
gmaxwell: speaking of, the -mintxfee behavior change may be worth a quick discussion.
< sipa>
cfields: the -paytxfee change you mean? :)
< sipa>
(too many fee parameters...)
< petertodd>
gmaxwell: oh! that's a good point!
< cfields>
sipa: er yes
< morcos>
i think we just bungled not more clearly announcing the change in semantics for paytxfee
< morcos>
surprising none of us flagged that as important at the time of the PR... which does raise another issue, we should have a checklist of things to revisit before release
< gmaxwell>
Did we know we really changed them? my view on the history was that it was changed to not round a long time ago, but another bug covered it up. That bug was fixed, and no one realized an announcement was needed.
< morcos>
multiple times now we've said, ok we'll just need to fix that before release, and then forgotten or almost so
< morcos>
gmaxwell: oh perhaps?
< Luke-Jr>
morcos: well, the change in behaviour is definitely correct
< gmaxwell>
I'm not sure that even if I realized it was a change I would have put "fee computation more accurate" as high importance-- since mining priority was changed to be precise a really long time ago. (0.6?)
< sipa>
morcos: when i saw that discussion, i remembered the "fPayAtLeastCustomFee" global and associated problems, but I don't think I ever realized that that global and its default value equal to true was ever released
< gmaxwell>
yea, I saw that fix but don't think I realized that it was ever in a release. When sipa asked me about paytxfee yesturday I told him it was changed to be accurate forever ago.
< gmaxwell>
So I think thats the sequence of errors here.
< gmaxwell>
A checklist would be useful, though I don't know if it would have saved us here.
< sipa>
so what i think happened is that at some point we switched the mining code to be per byte instead of per kb, later that global was introduced which implicitly retained the behaviour of "rounding up to 1000 bytes for fee calculation" even though the rest of the code was updated to be per byte, and only now, with the global going away, we actually get the accurate change
< gmaxwell>
asking people to document if a bug being fixed was ever released might have helped.
< morcos>
yeah , a checklist on changing behavior of any options or rpc calls being properly documented
< morcos>
another example is -maxsigcachesize
< sipa>
and i expect that people who made these changes were aware of it, as they updated the rpc tests accordingly, but not review
< morcos>
i pity the poor fool who has that set to 100000
< gmaxwell>
you don't have 100 gb ram?
< Luke-Jr>
ideally we should probably do the release notes in the PR itself
< morcos>
i'm not sure how many people would catch all these warnings in the 2 foot think binder of release notes, but its still good to have them
< gmaxwell>
I don't think it was a big deal here, but it could have been one.
< sipa>
well if we'd have warning for unknown options, we can just switch to a practice of renaming them whenever their meaning changes
< CodeShark>
make sure to say "WARNING" first so it's searchable :)
< btcdrak>
yeah I've been meaning to suggest we add at least brief release note documentation in PRs
< sipa>
btcdrak: i always do (or try to...)
< gmaxwell>
okay, we're going on a tangent.
< sipa>
going on a tangent is a sin
< gmaxwell>
Anything more to say about backlog before we move to talk fee filter?
< morcos>
oh no
< CodeShark>
no trig puns
< sipa>
CodeShark: i co-sign that
< gmaxwell>
My sides hurt.
< btcdrak>
sipa: can you cosign this?
< Luke-Jr>
lol
< Luke-Jr>
♥ sipa
< sdaftuar>
so how about that fee filter
< gmaxwell>
#topic feefilter
< morcos>
it seems to work pretty well
< gmaxwell>
Feefilter is awesome. I have not yet run it.
< Luke-Jr>
sorry, I need to run.. I think feefilter at least needs some kind of "mode" for things like "how do we measure size" etc, but not a huge deal
< morcos>
i mentioned in another context, it reduces tx send and rx bandwidth by around 40+%
< gmaxwell>
thats fantastic.
< morcos>
Luke-Jr: I'm basically of the mindset that we don't introduce complication until we need it
< morcos>
its totally optional, so no reason not to replace it later with a more generic one if we ever bother implementing
< gmaxwell>
We will not run out of message types, so we could introduce a modefilter later. I support that thinking.
< morcos>
it seems to me the message type is the version, yep
< gmaxwell>
I expect the way relay works to change substantially in the next couple years; so we should probably not overdesign here.
< morcos>
i need to do a trivial pr rebase, but i guess it just needs more review
< morcos>
i'm not sure what there is to discuss
< gmaxwell>
Okay, I will test and review. Thanks for working on this.
< morcos>
sure
< gmaxwell>
#action Test and review fee filter. Morcos reports unicorns and rainbows result.
< paveljanik>
great!
< morcos>
well all depends on your test setup i guess.. :)
< gmaxwell>
#topic CPFP mining
< gmaxwell>
sdaftuar: hows it going?
< sdaftuar>
it's awesome.
< sdaftuar>
i've been running live for the last two days
< btcdrak>
The PR number is #7600
< sdaftuar>
comparing existing mining algorithm to new one
< sdaftuar>
every ~128 tx's or so
< sdaftuar>
looking at the last call to CNB before a block is found, i see a 72% increase in fee/block on the last 144 data points
< gmaxwell>
I believe it should be making some pretty significant differences in selection from what I've seen. A number of users of OTHERBRAND wallet that has no fee estimation and always spends unconfirmed change seem to frequently produce chains of very low fee, very high fee (after realizing they needed more fee from the first tx).
< morcos>
72% ?!?!??!
< sdaftuar>
that could obviously be due to a small number of tx's that aren't getting mined for extended periods
< sdaftuar>
but geez we need this deployed, i think
< btcdrak>
amazing
< sipa>
sdaftuar: i believe that test would result in an exaggerated result
< gmaxwell>
the effect is likely exagerated due to the pattern I just described: the human controlled fees are exagerating the needed increase.
< sipa>
sdaftuar: as you're not actually creating blocks on the network with the new CPFP algorithm, i guess?
< sdaftuar>
yep
< sdaftuar>
correct
< sdaftuar>
so if a tx stays around for a day, and isn't selected by the old code, you'd count it over and over
< sipa>
sdaftuar: meaning that in a real setting, those "better" transactions would be mined once and cleaned up, reducing the effect for later blocks
< sipa>
right,
< sdaftuar>
correct
< sipa>
sdaftuar: how about performance?
< sdaftuar>
so there are three areas of performance to consider
< sdaftuar>
one is the additional work of the mempool to keep the index
< sdaftuar>
another is the part of CNB before TestBlockValidity is called
< sdaftuar>
and the last is the time TestBlockValidity takes (much larger than the rest of CNB, which is why i think it makes sense to split it out)
< gmaxwell>
(whom ever makes the lay summary, please don't report 72% increase as what CPFP does; in consideration of sipa's above point about N-fold counting)
< sdaftuar>
the mempool work isn't really a steady state increase, the concern i think is in what happens when a block is connected
< sdaftuar>
because then we have to update all the scores for every in-block transaction's descendants
< morcos>
gmaxwell: also the previous number he reported to me was 1%.. :)
< sdaftuar>
(when you add a tx to the mempool, you statically count its ancestors once, so that's basically negligible additional work)
< sdaftuar>
so i timed that extra delay in mempool.removeForBlock
< morcos>
but i think it is a good point, that if the increase is sometimes very big, its important for miners to take it... presumably the average increase wouldn't ever be much different from 0, as we don't see txs permantely residing in mempool
< sdaftuar>
and reported some numbers in #7594
< sdaftuar>
looks like what i saw was an increase from an average of 10.9ms to 11.2ms
< sdaftuar>
that was on half a month's data from october
< sdaftuar>
er 10 days i guess actually
< sdaftuar>
so i figure that's negligible enough to not really worry about, especially because if we really cared, we could make block relay happen while the mempool was still being updated, though it'd take some work
< gmaxwell>
do we worry that CPFP's utility is compromised without package relay? -- I guess these measurements suggest its not.
< sdaftuar>
moving on to CreateNewBlock's performance:
< sdaftuar>
vast majority of CNB's total time is taken up by TestBlockValidity
< CodeShark>
sorry to interrupt - we only have 8 minutes and I wanted to discuss segwit
< sdaftuar>
somehow, TBV is slightly faster using the new code than the old one. i haven't dived into it, but my guess is that maybe it's faster to look up mempool inputs than pcoinsTip inputs?
< sdaftuar>
that speed increase is actually larger than the small hit to performance on the rest of CNB, so it's actually faster in total. anyway numbers are in the PR #7600
< morcos>
gmaxwell: i don't see that as a big concern... i think it'll likely become common practice to avoid fees so small that they get evicted unless its actual spam. and CPFP will be useful for when you guess wrong on getting confirmed quickly
< sdaftuar>
i think this is a clear win
< gmaxwell>
sdaftuar: it sounds great, what now do you think we need to do to move forward?
< sdaftuar>
review! i broke the work into 3 PR's for review. one just adds the ancestor feerate index to the mempool (7594)
< gmaxwell>
morcos: I guess thats one upside over the overly large mempool default size.
< sdaftuar>
another is by morcos, which refactors CNB
< sdaftuar>
and then 7600 builds on both with the change to CNB
< morcos>
#7598
< gmaxwell>
#action whip people into wroking on review for CFPF 7594 / 7598 / 7600 it's nicely broken up.
< gmaxwell>
Can we segwit for CodeShark?
< CodeShark>
lol
< gmaxwell>
#topic segwit status
< CodeShark>
we had a fork a few days ago
< sipa>
i haven't had time to investigate
< sipa>
my hope is that it is caused by miners running older versions of the code
< sipa>
and not something else
< gmaxwell>
Time for science then.
< CodeShark>
that's most probable - but we haven't narrowed down the conditional that actually caused it
< sipa>
i was planning on doing a segnet4 very soon, but we'd need to understand what's causing this first
< morcos>
well is there anyone stuck on the short fork?
< CodeShark>
I think there might still be a few
< morcos>
seems like would be helpful to know what errors they have and what code they are running
< cfields>
hmm, i'd be interested in taking a look there. sipa: any helpful references/context ?
< gmaxwell>
might be useful if regtest networks put their git build info in their version numbers. awful for privacy... but would be useful here.
< sipa>
cfields: CodeShark probably knows more
< gmaxwell>
(e.g. a chainparam to put that info in the subver)
< cfields>
ok. will ping CodeShark after
< CodeShark>
I think the offending block was something like 22130
< CodeShark>
or 22132
< CodeShark>
or somewhere around there
< gmaxwell>
okay So-- sounds good, a fleet of flying monkies will contemplate the segnet fork. Posting forked IPs in the segwit IRC channel might get someone's attention.
< gmaxwell>
Sorry we didn't get to all the topics.
< morcos>
we still need tests for the soft fork BIPS right
< morcos>
and 7187 still needs to be merged as well..
< btcdrak>
morcos: I'm waiting on the python tests from sdaftuar for #7187
< gmaxwell>
sdaftuar: if CPFP appears to be moderately stable, we might consider asking a moderately large miner to run it (in parallel to other stuff); it would have most of it's usability benefit for the network if only one moderately large miner was running it.
< sdaftuar>
gmaxwell: yeah i was wondering if any miners might be set up to test the new code using their production parameters at least? so that we can measure performance in production settings and know we haven't missed anything
< sdaftuar>
i thought it might make sense to wait until it was merged into master to ask someone to do that
< gmaxwell>
sdaftuar: assuming that the surrounding enviroment is sufficently fail safe, even if it's a crash problem then it should be safe to try. but also no harm in getting some more maturity under its belt first.
< gmaxwell>
The only reason I suggested it is because there are at least a few users whos delays could be avoided by it.
< sdaftuar>
gmaxwell: that sounds reasonable to me. do you have someone in mind to reach out to, or should i send out an email to the -dev list?
< gmaxwell>
sdaftuar: I have someone in mind.
< sdaftuar>
gmaxwell: cool, feel free to put them in touch with me