< gmaxwell>
people are misinterperting the gihub license, the things people are objecting to are related to moral rights, not copyright, moral rights largely don't exist in the US.
< wumpus>
at some point, github is going to do something really stupid and we'll see an exodus of projects, just like what happened to sourceforge, and we'll have to move our main development hub somewhere else. But this seems largely based on misinterpretation and very avid attempts at "reading between the lines", at least it seems to me...
< gribble>
https://github.com/bitcoin/bitcoin/issues/8775 | RPC refactoring: Access wallet using new GetWalletForJSONRPCRequest by luke-jr · Pull Request #8775 · bitcoin/bitcoin · GitHub
< wumpus>
luke-jr: adding it to my list...
< wumpus>
indeed seems important to get that merged now after the 0.14 fork
< wumpus>
but I need to go over it in detail
< luke-jr>
#7533 would be nice to get over-with (needs rebasing often), but is lacking review (8775 has a decent amount already)
< gribble>
https://github.com/bitcoin/bitcoin/issues/7533 | RPC: sendrawtransaction: Allow the user to ignore/override specific rejections by luke-jr · Pull Request #7533 · bitcoin/bitcoin · GitHub
< bitcoin-git>
[bitcoin] ian-kelling opened pull request #9903: Docs: add details to -rpcclienttimeout doc (master...docs-client-timeout) https://github.com/bitcoin/bitcoin/pull/9903
< bitcoin-git>
[bitcoin] laanwj opened pull request #9904: test: Fail if InitBlockIndex fails (master...2017_03_test_check_blkindex_result) https://github.com/bitcoin/bitcoin/pull/9904
< ryanofsky>
jonasschnelli: something about shared_ptr?
< jonasschnelli>
ryanofsky: I think i could solve it...
< * BlueMatt>
would like to discuss the lack of unicorns 'round here
< sdaftuar>
topic suggestion: any open items for 0.14?
< wumpus>
BlueMatt: +1
< wumpus>
any new issues with rc3?
< gmaxwell>
I have an issue.
< gmaxwell>
It isn't released yet. :P
< sipa>
ha.
< wumpus>
hehe
< BlueMatt>
yup
< BlueMatt>
rc3 was posted yesterday, so release is next tuesday if nothing else comes up, I believe?
< sipa>
seems reasonable
< gmaxwell>
I saw some positive comments on Bitcoin talk.
< BlueMatt>
dont we normally do a week after rc?
< wumpus>
yup, that's how it useually goes yes, BlueMatt
< MarcoFalke>
#action plan to release next tuesday if nothing else comes up
< morcos>
i'd like to briefly discuss the timing of merging #9602.. b/c assuming we are going to do it, it would be nice to get it out of the way so its not a rebase/review nightmare. i also have a ton of fee estimation changes built on top
< sdaftuar>
+1 for merging sooner rather than later
< wumpus>
well if it is ready for merge it should be merged
< sdaftuar>
i'm also working on some mining tweaks that i'd rather just build on top of 9602
< BlueMatt>
wumpus: I believe it needs review and morcos is asking for fast-review because its a pita to rebase constantly
< morcos>
wumpus: oh ok, i just wanted to coordinate so people were reviewing at the same time frame... thats when the rebases get annoying, when ppl have reviewed different versions. and now it seems people have started reviewing
< morcos>
so anyone else that is interested, sooner is better than later
< gmaxwell>
Sounds fine to me, a related subject is that there are a number of other features (like multiwallet) that just missed 0.14 which we should try to get in early rather than later.
< wumpus>
yes
< BlueMatt>
yes, I imagine luke's dont-use-pwalletMin thing is a pita to rebase as well
< BlueMatt>
that would be #8775
< gribble>
https://github.com/bitcoin/bitcoin/issues/8775 | RPC refactoring: Access wallet using new GetWalletForJSONRPCRequest by luke-jr · Pull Request #8775 · bitcoin/bitcoin · GitHub
< gmaxwell>
it's also an issue that if these things drag on until late in the cycle they'll miss again.
< wumpus>
right, so all review multiwallet pulls
< wumpus>
#8775 should probably go in first
< gribble>
https://github.com/bitcoin/bitcoin/issues/8775 | RPC refactoring: Access wallet using new GetWalletForJSONRPCRequest by luke-jr · Pull Request #8775 · bitcoin/bitcoin · GitHub
< MarcoFalke>
They don't conflict according to git merge, so no strict order required
< gmaxwell>
It might be useful, not in this meeting, but for people to think about what they'd like the big features of 0.15 to be and make sure we make progress early enough on those things so that they can actually be there. :) I feel like there were things that at least I personally wanted in 0.14 but didn't give them enough attention until too late.
< morcos>
1 major feature thats in the back of my mind, but a bit complicated so might require some discussion as to whether we want it and when is automated fee-bumping
< wumpus>
well at least there should be time for features again in 0.15. 0.14 time was mostly spent on segwit
< sipa>
famous last words :)
< sipa>
but i agree
< gmaxwell>
morcos: I would replace 'automated fee bumping' with 'precomputed fee bumping' E.g. when you sign, presign all the bumps, locktimed... so they don't interfear with the wallet encryption.. and could even be handed off to someone if you go offline.
< wumpus>
well I don't know of any large upcoming softfork projects monopolizing everyone at least?
< gmaxwell>
wumpus: everything like that is on hold for segwit!
< wumpus>
so hopefully for 0.16 again then :)
< wumpus>
anyhow, it means for 0.15 we can focus on software features instead of network/consensus features
< morcos>
gmaxwell: do we not have any suggested SF's that aren't built off segwit in the queue? lets take advantage of BIP 9!
< sipa>
raise min difficulty, optional utxo commitments, ...
< gmaxwell>
on that subject, we should reconsider some things around how segwit works: that we won't mine once segwit is active without the flag set, and we don't set the versionbit without the flag... Both of these are foolish IMO.
< sipa>
oh, by flag you mean whether the GBT client indicates support?
< gmaxwell>
by 'don't set the versionbit without the flag-- if the GBT client doesn't signal segwit support we don't set the BIP 9 bit. Which is stupid, since we'll happily enforce the rules.
< sdaftuar>
we do that so that segwit can't activate without the miners actually mining segwit transactions
< BlueMatt>
sdaftuar: I'm not too worried about that
< gmaxwell>
sdaftuar: yea but I think that is an error. So what if they don't? Then there is just less capacity for segwit txn initially until they upgrade, and they'll lose out on fees.
< BlueMatt>
I prefer for miners to be able to mine and just lose out on fee revenue than stop mining
< wumpus>
#action review and merge #8775 and #9602 as soon as possible, they are prone to turning into rebase/merge nightmares
< gribble>
https://github.com/bitcoin/bitcoin/issues/8775 | RPC refactoring: Access wallet using new GetWalletForJSONRPCRequest by luke-jr · Pull Request #8775 · bitcoin/bitcoin · GitHub
< sdaftuar>
gmaxwell: my concern (before we got to the point we're at now) was that segwit could have activated with 0 miners mining
< sdaftuar>
and then mempools could be attacekd with transactions that wouldn't confirm
< morcos>
gmaxwell: agreed, as long as we know that SOME miners are mining them.. which seems we're at that poin tnow
< sdaftuar>
perhaps now we can relax it
< gmaxwell>
sdaftuar: Yes, I agree. That would be silly. But we're not there now. :)
< gmaxwell>
I didn't protest at the time because of that. (well actually on the stops mining thing, I thought we fixed that)
< sipa>
suggested topic: CNB calling TBV or not, and CNB caching
< gmaxwell>
(My misunderstanding was that I thought we stopped mining without the gbt-flag entirely once the code had support for it.. and I saw that wasn't the case)
< gmaxwell>
We need to get TBV out of the critical path. I do not really agree with lightswords view on it though-- I think it's important that we have some process that tests that blocks were handing out to mine. It does not need to be in the critical path.
< wumpus>
#topic CNB calling TBV or not, and CNB caching
< sipa>
i believe that what the code does for an actual miners doesn't matter
< morcos>
gmaxwell: sipa: i think the downside of leaving it in the critical path is an extra 150ms of mining with an empty block
< sipa>
yes, that's obvious
< sipa>
and there are various solutions
< gmaxwell>
morcos: or worse.
< sipa>
an easy one (just get rid of the test)
< sipa>
or a hard one (background validation, caching, ...)
< gmaxwell>
The challenge with backgrounding it is that test holds cs_main for a long time.
< morcos>
hmm, i guess i was wrong
< gmaxwell>
I have suggested making the validation it does interruptable.
< BlueMatt>
gmaxwell: I have suggested that many times :)
< morcos>
if you want to build on a prior header, you can't have TBV in the critical path, b/c you might not be even able to do it if you dont' have prior block
< gmaxwell>
E.g. an atomic checked between each transaction which can abort a test validation. This would also be useful for the block proposal validations.
< gmaxwell>
then an incoming block will set the atomic and then block on acquiring cs_main.
< sipa>
morcos: well building on a prior header is easy... i'm sure we can build an empty block that's valid
< gmaxwell>
the background thread would just go to sleep and try again later. Block proposals would just sleep for a bit then try again (while the RPC caller waited)
< morcos>
yes but we have to have a way of skipping TBV on that block don't we? or some really hacking version that calls TBV on a fake chain active
< sipa>
an alternative is just having a background process that every so often runs CNB and caches the result
< sipa>
and then validates it after storing in the cache
< gmaxwell>
Well my thought would be we'd just have a background loop, running even if you don't mine, that TBVs once a minute or so.. and it can get interrupted. and if its interrupted it just gives up until the next minute.
< morcos>
it only needs to be out of the criticial path when there is a new tip... when its just updating the block, it doesn't matter if you wait for it
< sipa>
then GBT can check whether the cached block still builds on top of the best header, and if not, return an empty block
< gmaxwell>
okay if doing what sipa suggests a minute is a bit slow!
< sipa>
and a new tip would obviously wake the background thread
< gmaxwell>
but sounds like we were thinking of similar-ish things.
< sipa>
however, we wouldn't want such a background CNB for normal nodes
< gmaxwell>
I dunno, at a low enough frequency it would be fine and would create a lot more in-field testing.
< morcos>
the good thing about going down that road is you can be smarter about waking the CNB thread to wait until you have new high fee txs that are at least n seconds old or what have you
< sdaftuar>
morcos: are you going to PR something that calls CNB to keep pcoinsTip warm?
< sipa>
and it would keep the sigcache warm with what we expect to be mined
< morcos>
running TBV on slow hardware over and over is probably bad
< gmaxwell>
One thing I suggested previously is that calling GBT pump the refresh of the cache... so it runs more often if you're calling GBT than otherwise.
< sipa>
gmaxwell: seems reasonable
< morcos>
sdaftuar: i don't remember.... but i think the version i had, only ran it once after a flush
< sipa>
this not only takes TBV out of the critical path, but also CNB
< gmaxwell>
morcos: once a minute? it wouldn't be horrible. Once every 10 minutes? 20 minutes? I think there is a number where its harmless and we would avoid having code that runs only on a very small number of nodes thus gets inadequate operaiton to expose bugs.
< morcos>
honestly i think a more important direction to go would be to start by replacing GBT with a better framework
< morcos>
and then making that work optimally
< sipa>
morcos: i think this may be orthogonal work
< gmaxwell>
I think this is more or less orthorgonal to GBT. In that anything else will still need to create a block template. :)
< sipa>
it'd just be a wrapper around CNB
< morcos>
i agree its mostly orthogonal.. but one still probably has to occupy our attention first
< sipa>
well GBT replacement needs a lot of external discussion
< gmaxwell>
Well there are design considerations in the 'better thing' that hinge really on how fast CNB is ... and the spectrum of options we want to hand out when we can't give an answer fast.
< morcos>
gmaxwell: hmm.. yes... perhaps what i mean is we should design the better thing so it informs what we want out of CNB/TBV. and document the design so we dont' forget.. even if we don't do it yet
< gmaxwell>
I think right now I don't have a lot of appetite for major inititavies that we can't just do within the project. But if someone wants to undertake a big coordination effort, that shouldn't slow them down.
< gmaxwell>
Okay, well an expirement is not something we need permission for... :)
< morcos>
ok no argument here
< morcos>
i guess this all hinges on making the TBV code (and maybe even CNB) interruptible
< sipa>
i think that's relatively easy
< morcos>
yeah thinking about it, they both are pretty trivial
< gmaxwell>
Tightly related to this question is the question of bypassing validation for things in our template. As you may be aware some people have been using in-mempoolness as a proxy for validity. This seems rather sketchy to me, though it should be a pretty substantial latency improvement. The assumption that makes ditching TBV okay also makes doing that okay, I believe. And there is an open alternat
< gmaxwell>
ive which is potentially more safe, if a transaction is in our template cache for this height, which we've already verified, then its validation could be skipped.
< morcos>
i'm basically opposed to that
< sipa>
relying on the template-validation-cache to be correct seems less risky than relying on the mempool itself to be correct
< gmaxwell>
And if it results in users not using this software but software written by people who don't even know enough to oppose that?
< sipa>
but it does make TBV consensus critical
< morcos>
i do think we could do these things while waiting for validation
< gmaxwell>
In any case, just a thought.
< luke-jr>
TBV is already consensus-critical, more or less
< luke-jr>
the code for it anyhow
< gmaxwell>
(that using a tested template is safer than the mempool)
< sipa>
luke-jr: well it uses most of ConnectBlock, which definitely is consensus critical
< morcos>
e.g. get a new block from the network... -> assume valid -> mark all txs in mempool as potentially used -> CNB from remaining -> have not yet validated new tip or TBV new template, and if we find a block, so be it...
< sipa>
luke-jr: that overlap is what makes it less risky
< gmaxwell>
morcos: that is also interesting.
< gmaxwell>
so fast for mining but nothing else.
< morcos>
gmaxwell: i guess its not QUITE that easy
< gmaxwell>
morcos: but in that case you would extend an invalid block, bad.
< morcos>
gmaxwell: yes but only for a couple hundred ms... you still immediately validate and switch work if there was a problem
< gmaxwell>
(very bad to have miners extending invalid blocks, even for relatively brief intervals, massively amplifies risk for lite clients-- especially since devices may stay on old work for tens of seconds)
< sipa>
plan: 1) make TBV interruptible 2) add background thread that caches CNB results 3) make GBT return an empty block if the cache does not build on your tip
< gmaxwell>
(and I got flamed to death when I suggested a singaling mechenism for lite clients to ignore confirmations constructed that way)
< gmaxwell>
sipa: that plan sounds good.
< morcos>
well we can't make our plans based on what you get flamed for can we? :)
< BlueMatt>
gmaxwell: I'm more a fan of not relying on validation being fast - mine empty blocks for the 100ms it takes to get a blocktemplate, and relay blocks during validation
< BlueMatt>
problem solved :)
< sipa>
agree
< sipa>
where did you get flamed?
< gmaxwell>
BlueMatt: you still need to validate a block before extending it. To not do so is toxic to lite clients which strongly trust that you have.
< gmaxwell>
bitcoin-dev
< morcos>
sipa: ha ha ha, that was a funny
< BlueMatt>
gmaxwell: oh? every time I've discussed it with anyone in person they're like "yea, do that, sounds good" :)
< gmaxwell>
I wrote a spec for signaling that you've not validated the prior block. So that lite clients could just ignore those blocks as confirmations.
< sipa>
morcos: it was an honest question - i can remember gmaxwell bringing it up in here a few times, i didn't know there was more to it
< gmaxwell>
I wrote a BIP, draft-maxwell-flagverify
< BlueMatt>
gmaxwell: and I think we should implement that when we go to return empty blocks :)
< BlueMatt>
I'm happy to go implement it in lite clients, too
< gmaxwell>
I also think the rise in fees is such that it's increasingly less interesting to produce empty blocks.
< BlueMatt>
true, but 12.5 BTC is sitll > 0 BTC :P
< luke-jr>
should GBT return partial blocks, as the background thread fills it?
< gmaxwell>
lightsword isn't here now it seems but previously he has pointed out that returning an empty template is often bad because the downstream software stack doesn't know to try again immediately and so will mine on it for a long time.
< BlueMatt>
there is just more pressure to limit the time you're spending mining empty blocks :)
< BlueMatt>
Lightsword: yes he is
< luke-jr>
gmaxwell: what Eloipool does to solve that, is return a longpollid which is triggered as soon as the full template is available
< gmaxwell>
BlueMatt: yes but that isn't the tradeoff. Assuming you mine on the template for --say-- 30 seconds, it's 13.4 BTC expected (13.5 - 100ms of mining) vs 12.5. or whatnot.
< sipa>
well, we could optionally also make GBT return a partial block and/or block until the full block is available (but not yet TBVed)
< gmaxwell>
e.g. current fees pay for 100ms of delay easily.
< BlueMatt>
gmaxwell: oh? I believe most pools can push a "hard update" or whatever its called
< BlueMatt>
that will force the asic to switch
< sipa>
moving the actual construction to the background won't hurt
< BlueMatt>
there is a flag in stratum for it
< BlueMatt>
(to flush workqueue in asic)
< gmaxwell>
BlueMatt: for some hardware retargeting causes misbehavior/loss of performance.
< gmaxwell>
In any case, it isn't so simple.
< BlueMatt>
fair
< BlueMatt>
I assume most modern hardware supports it, though
< BlueMatt>
given that most pools do this
< BlueMatt>
incl pools run by the hardware mfgrs
< gmaxwell>
Do not assume that.
< BlueMatt>
ok, fair
< sipa>
we're running out of time
< sipa>
any other topics?
< gmaxwell>
quick, empty messages
< sipa>
< luke-jr>
< wumpus>
< gmaxwell>
< luke-jr>
inb4 trolls use this as proof we obey gmaxwell
< gwillen>
< BlueMatt>
lulwut
< wumpus>
#topic
< sipa>
it's "lolwut", BlueMatt.
< BlueMatt>
lulzwutz
< wumpus>
cleared the topic, too, now we can cleanly exit the meeting
< gmaxwell>
We should appoint a subcommittee to investigate the spelling of lolwut/lulwut.
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Mar 2 19:56:59 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< Chris_Stewart_5>
What do the acronyms CNB and TBV stand for?
< sipa>
CreateNewBlock, TestBlockValidity
< Chris_Stewart_5>
Thanks.
< Lightsword>
gmaxwell, morcos, sipa, one other approach is to use a more strictly performance optimized version of CNB for the first block template that doesn’t do all the fee maximizing calculations and just takes the top of the mempool
< gmaxwell>
Lightsword: the fee stuff I think takes pratically no time at all.
< gmaxwell>
because it's all built from efficient indexes.
< gmaxwell>
An empty template can be faster because it can be constructed without the prior block having removed its transactions from the mempool... because it needs ~no hashing to construct, etc.
< gmaxwell>
back in the days of priority constructing a template was objectively slow. Now it's just traversing a precomputed index pretty much.
< Lightsword>
gmaxwell, do we loop over the index more than once?
< sipa>
no
< sipa>
but it's a big data structure, traversing indexes that are spread out over memory can take time
< sdaftuar>
fyi i've been benchmarking today
< sdaftuar>
it's a bit slower now than it was last time i benchmarked, so i'm going to investigate
< gmaxwell>
Interesting. previously and my expectation was that it was just a couple milliseconds, and that we spend more time seralizing the data to json.
< sdaftuar>
yeah it used to be like 7-10ms of transaction selection, followed by ~35-45ms of TBV, or something like that
< sdaftuar>
but on my runs today on data from December, it's been more like 30-40ms for each of those parts, maybe a bit more
< gmaxwell>
That is interesting.
< sdaftuar>
Lightsword: depends on the state of the mempool. i think it's possible that if there are more chained transactinos now than the last time i analyzed, it could slow things down
< sipa>
Lightsword: GetMemPoolChildren just returns a reference to a precomputed list of children
< sipa>
i wonder if BOOST_FOREACH makes a copy of it, though
< sdaftuar>
oops, i was thinking of CalculateMemPoolDescendants() or whatever :)
< Lightsword>
sdaftaur, are there any other calls similar to that which could be potentially skipped on the first run through the mempool?
< sdaftuar>
Lightsword: potentially, yeah we could consider dropping the re-sorting of descendants of selected transactions
< sdaftuar>
i'll add that to my list of things to consider
< Lightsword>
my thoughts are that the CNB first call after a block change we can skip descendant calculations entirely and only add transactions that have confirmed inputs
< gmaxwell>
well you could also only take the highest feerate transactions that capture something like 90% of the available fee, and potentially return a short template that will be faster to seralize and send.
< gmaxwell>
but you will still need to wait then for the prior block to remove transactions from the mempool... which is not blindingly fast.
< Lightsword>
how long are you seeing full GBT serialization take?
< cfields>
dammit, I managed to completely miss the meeting
< Lightsword>
cfields, we decided you get to rewrite GBT :P
< cfields>
heh
< cfields>
I blame wumpus :p. i was distracted by the fs abstraction and lost track of time.
< wumpus>
cfields: Thanks! heh myself I got stuck on a very weird miscompile issue with clang: https://github.com/NuxiNL/cloudabi-ports/issues/30 (likely nothing we have to worry about for our builds, except that we shouldn't be enabling safestack for now)
< wumpus>
jonasschnelli: cool
< jeremyrubin>
wumpus: does they cloudabi stuff exist locally as a sandbox you then run bitcoin in or do you compile it in?
< wumpus>
jonasschnelli: : that web interface is pretty neat
< cfields>
wumpus: nice responsiveness from them
< wumpus>
jeremyrubin: the normal way is to use "cross-compilation", which are available for various platforms. I don't think the compilers themselves run in it yet
< wumpus>
cfields: yes absolutely!
< cfields>
wumpus: i wonder if this effort would help with enabling sandboxing for macos as well
< cfields>
similar restrictions, i would assume
< wumpus>
cfields: I think so
< jeremyrubin>
I guess I was asking because it would be easier to run untrusted binaries
< jeremyrubin>
e.g., you run the launcher program that you know is secure on your machine, which passes the FD to the binary that's marked as unable to get any new capabilities
< wumpus>
jeremyrubin: yes, that's what you can use it for
< wumpus>
jeremyrubin: the program cannot do anything that you haven't allowed to it, not look around the filesystem, not look at hardware identifiers, not connect to the network, etc
< cfields>
jonasschnelli: very cool
< jeremyrubin>
wumpus: but only if you know you compiled it? If someone else compiled it, they could just not link against cloudabi
< jeremyrubin>
random question:
< jeremyrubin>
How important is it that CPubKey::Verify takes a hash?
< jeremyrubin>
e.g., could it be any (padded to 32 byte) data safely?
< cfields>
jeremyrubin: i assume it sets a libc dep, or even elf identifier
< cfields>
jeremyrubin: for what purpose?
< jeremyrubin>
cfields: looking at checksigfromstack implementation
< jeremyrubin>
cfields: I don't think it should include a digest
< jeremyrubin>
yeah I can't think of how it could be unsafe (except maybe unsafe for signers if they sign 0 or something)
< cfields>
jeremyrubin: er, we surely don't want that calculation happening at the pubkey layer?
< bitcoin-git>
bitcoin/master 7ed143c Russell Yanofsky: Add test for CWalletTx::GetImmatureCredit() returning stale values....
< bitcoin-git>
bitcoin/master f7ec7cf MarcoFalke: Merge #9359: Add test for CWalletTx::GetImmatureCredit() returning stale values....
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #9359: Add test for CWalletTx::GetImmatureCredit() returning stale values. (master...pr/immature) https://github.com/bitcoin/bitcoin/pull/9359
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #9905: contrib: Move second sha512 check to the end (master...Mf1703-gh-merge) https://github.com/bitcoin/bitcoin/pull/9905
< arubi>
does anyone know, was there ever a writeup \ -dev thread about witnessV1\sighashV2 from jl2012's mastV3 fork I see it enables a lot more ways to sign transactions. I'd appreciate a manual :)