< BlueMatt>
cfields_: the only logic in the added node stuff that I think is really useful to keep around is the logic around the connecting thread
< cfields_>
BlueMatt: yes, that's preserved.
< BlueMatt>
cfields_: eg if you change an added dns name's ip it should connect to one of the new ones
< BlueMatt>
though if you change the getaddednodeinfo stuff too much it'll be useless?
< cfields_>
BlueMatt: but as far as rpc is concerned, I think that logic should kinda be a black box. It should only be reporting which addresses you've requested to connect to, and which are currently connected
< BlueMatt>
well how does it know which are currently connected without a dns lookup?
< BlueMatt>
unless you keep a cache from dns -> ip that is updated every time the connect thread runs through its loop
< cfields_>
BlueMatt: node should preserve the original dns as well as the resolved
< BlueMatt>
"original"?
< BlueMatt>
oh, you mean CNode?
< cfields_>
yea
< BlueMatt>
no, thats not sufficient...if you change the ip of a dns name, then the rpc will still report that you're connected even if you're not connected to the new name
< cfields_>
BlueMatt: that's my point though, doing a new resolve won't necessarily give you the same results. For ex, if you add a seed node, which has a very short ttl, you won't find a match if you resolve again a few min later
< BlueMatt>
this is confusing because now the background thread is doing something that is inconsistent with what the rpc will show you
< BlueMatt>
ehh, I'm less concerned about that, but if thats your concern then it seems the only way is for the background addnodes thread to keep a cache of its resolutions
< cfields_>
BlueMatt: ah i see your point
< BlueMatt>
I guess I see the RPC as answering the question "Am I currently connected to what the background thread would connect to if it ran right now?"
< cfields_>
BlueMatt: erm wait no, that doesn't follow. If i addnode foo.com, it resolves to 1.2.3.4, connects, then changes to 5.6.7.8, i'd say it's reasonable for rpc to report that foo.com is connected.
< cfields_>
yes, i see now that it's unclear.
< BlueMatt>
yea, I prefer the RPC to work the other way
< BlueMatt>
I'd say the RPC thread should report that it is not connected, because that would imply to the user that its going to try to connect
< BlueMatt>
which it is
< cfields_>
ah, there's the problem. i didn't get that logic right after all :)
< cfields_>
BlueMatt: thanks, understood now. I'll fix that up.
< jonasschnelli>
So ~2.5* slower then leveldb on the 4core 64bit linux machine I have testes.
< gmaxwell>
doesn't really make sense that reindex would be slower.
< gmaxwell>
since the sync is so much faster.
< gmaxwell>
must be some reindex specific bug.
< jonasschnelli>
gmaxwell: the sync is faster because it didn't touch the db (was dbcache=9000)
< gmaxwell>
jonasschnelli: I benchmarked lmdb sync last weekend with default db cache.
< gmaxwell>
it was _significantly_ faster than leveldb.
< jonasschnelli>
hmm...
< gmaxwell>
5 hours 5 minutes vs 8 hours 27minutes.
< jonasschnelli>
I'll do now a reindex with the same parameter on current master (leveldb).
< wumpus>
so normally a reindex takes 2h40 on that machine with leveldb with default params?
< jonasschnelli>
Hmm... the 2h40 was a sync from random peers. Not sure if I acctually did a reindex with leveldb. Doing now.
< jonasschnelli>
I mean these are not scientific benchmarks... I'm just playing around.
< wumpus>
compare reindex against reindex please :)
< jonasschnelli>
yes. I'll do now.
< wumpus>
sync from network can be faster than reindex in some cases
< gmaxwell>
do we still have the thing where signature checks run during reindex at all blocks?
< jonasschnelli>
I just expected the lmdb reindex to be faster then the leveldb random peer sync
< jonasschnelli>
But right. could be an performance issue in reindex
< wumpus>
(for that reason I tend to benchmark syncing from a fast local-network peer instead of reindex these days)
< wumpus>
gmaxwell: I don't know, I don't think I even know of that issue
< NicolasDorier>
wumpus: how is it possible ? as I thought reindex would do less work than a resync
< * gmaxwell>
checks the microphone.
< wumpus>
NicolasDorier: you'd expect so
< gmaxwell>
yes it appears we do.
< gmaxwell>
wumpus: I complained about it so much that I gave up complaining.
< wumpus>
NicolasDorier: the difference is that a reindex doesn't need to write to block files, just read, and does so linearly instead of haphazard
< wumpus>
gmaxwell: did you ever file a github issue?
< gmaxwell>
It's mildly hard to fix.
< gmaxwell>
lemme see
< gmaxwell>
and it looks still unfixed.
< wumpus>
please do file an issue if there is none, complaining on IRC always gets lost
< gmaxwell>
basically the reindex doesn't run headers first, so it doesn't have the headers ahead of the blocks
< gmaxwell>
thus if (pindexLastCheckpoint && pindexLastCheckpoint->GetAncestor(pindex->nHeight) == pindex) {
< gmaxwell>
doesn't work.
< gmaxwell>
and scriptchecks run on all blocks.
< jonasschnelli>
okay. Thats explains it.
< jonasschnelli>
We need to fix that. :)
< wumpus>
yes
< gmaxwell>
I'm looking to see if there is an issue. I believe there was.
< NicolasDorier>
wumpus: if reindex does not need to write to block file, why is it slower ? it does not make sense oO
< wumpus>
NicolasDorier: read what gmaxwell writes above
< gmaxwell>
secp256k1 made it so much faster that it was like fixing it... at least for a bit, chain growth is compensating. :)
< wumpus>
this is not a concurrent i/o issue but a validation issue, apparently, I also always assumed it had something to do with i/o but reasoning that through doesn't make a single bit of sense
< NicolasDorier>
oh
< NicolasDorier>
I understand, thanks
< NicolasDorier>
well, if you want to test performance, it is better to NOT have checkpoints right ?
< gmaxwell>
NicolasDorier: sure but it resulted in jonasschnelli doing an apples/oranges test.
< wumpus>
depends on what performance you want to measure, just compare apples against apples
< wumpus>
if you want to compare pure database performance, disabling validation makes some sense
< jonasschnelli>
gmaxwell: Yes. Right. Now started comparing Apple/Apples (reindex leveldb is running).
< jonasschnelli>
What we really want is a db benchmark script.
< gmaxwell>
in any case, we'd also like to get rid of checkpoints for script check bypassing, but what we'd replace them with would also need the headers... so...
< jonasschnelli>
script = test-suite thing.
< wumpus>
well it's easy enough to write a script that compares leveldb and lmdb (see e.g. the one in blockdb-troubleshoot that copies db-to-db), however what we want to measure is bitcoind's specific usage pattern not some dumb micro-benchmark - and that's almost impossible with an external script :)
< wumpus>
I agree though that a more reproducible kind of benchmark would be nice. In principle a reindex with disabled validation wouldb e an ok one I think
< jonasschnelli>
Right. That could work.
< wumpus>
(at least not anything that involves internet peers)
< jonasschnelli>
I think sync from a local peer is also not really representative.
< wumpus>
why not?
< jonasschnelli>
The local peer could do some other work during your test (like a backup or a heavy log rotate, etc.).
< gmaxwell>
if it's local you control that.
< wumpus>
well no I make sure it is completely available to the other node for downloading, it has no other network connections
< jonasschnelli>
Sure. But I'd prefer something that exclude network and another linux.
< gmaxwell>
bigger problem is that low latency peers will conceal pipelining failures.
< gmaxwell>
like, if you fetch one block at a time, a local peer will not be that slow, but then add a 50ms RTT and it's MUCH slower.
< gmaxwell>
of course, one can simulate real networks locally using dummynet.
< wumpus>
yes there it gets into really difficult territory
< jonasschnelli>
I think the reindex with disabled verification is probably better
< wumpus>
benchmarking the network code
< wumpus>
we're in 'benchmarking for dummies' here with the database :)
< gmaxwell>
jonasschnelli: depends on what you're testing.
< jonasschnelli>
For the db performance comparison
< wumpus>
for the db performance comparison using a reliable local peer is good enough - that won't be the bottleneck
< gmaxwell>
e.g. faults in the database handling can take the form of excessive seralization such that IO and network doesn't overlap; testing only with reindex will hide that.
< wumpus>
if you want to test pipelining in networking, well, good luck, there's stacks of books written about that :)
< gmaxwell>
so my point is that the reindex like test is informative, but not complete.
< gmaxwell>
wumpus: one can do a lot better just by testing with a more realistic setup.
< wumpus>
maybe we can find a phd student hehe
< wumpus>
gmaxwell: sure
< gmaxwell>
e.g. for the testing for the recent inv changes sipa and I are working on, I setup a 14 node topology.
< gmaxwell>
(with actual systems, though that test could have been done with dummynet and VMs okay, I think)
< wumpus>
tools that reproduce internet circumstances on a simluated network are great for that, absolutely
< jonasschnelli>
sipa: > nOldFee already is enough for all previously replaced ones, so you only need to add minTxRelayFee to account for everything. That's the logic used by the replacement code as well.
< jonasschnelli>
What if you need to add an input to fit the new fee?
< sipa>
yes?
< jonasschnelli>
bigger tx size, more fee required.
< GitHub61>
bitcoin/master 72c54e3 Wladimir J. van der Laan: Merge #7863: getblockchaininfo: make bip9_softforks an object, not an array....
< GitHub19>
[bitcoin] laanwj closed pull request #7863: getblockchaininfo: make bip9_softforks an object, not an array. (master...bip9-status-as-object) https://github.com/bitcoin/bitcoin/pull/7863
< sipa>
do we backport 7863 to 0.12?
< wumpus>
yes, it needs to be backported to 0.12 eventually
< wumpus>
if there is 0.12.1rc3 may as well be in it
< Luke-Jr>
I wonder sometimes if it should be named "bip9", but I can argue both ways, so I haven't brought it up.
< Luke-Jr>
probably not important enough to matter
< wumpus>
yes, feel no need to even have an opinion about it
< wumpus>
cfields: let's have a chat some time about how to handle out-of-tree builds (#7466), I'd really like that working, we have two solutions, but I guess neither is mergeable yet and I'm not sure how to continue
< wumpus>
(I'd really like to switch to a setup where the source code is on a different disk from the binaries)
< sipa>
wumpus: squashfs? :p
< wumpus>
lol would you compress the binaries or the source code?
< sipa>
oops, i mean unionfs
< wumpus>
could work I guess, but seems overkill, every project just supports out-of-tree builds except bitcoin core :)
< wumpus>
but unionfs does look interesting, could help in some other cases
< sipa>
hah, seems unionfs was the original used by Knoppix, since then there has been aufs (which was rejected by linux mainline, but the alternative overlayfs was merged)
< wumpus>
the main motivation really (besides maybe performance) is to have a clearer separation between stuff that needs to be backed up and stuff that could just be rebuilt if lost
< sipa>
maybe it would be interesting for me as well... whenever my laptop battaery dies while compiling bitcoin core (not surprisingly, that's the most common activity during which power fails), my git tree is corrupted
< GitHub62>
[bitcoin] laanwj closed pull request #7850: Removed call to `TryCreateDirectory` from `GetDefaultDataDir` in `src/util.cpp`. (master...issue7845) https://github.com/bitcoin/bitcoin/pull/7850
< GitHub48>
bitcoin/master 62a6486 Pavel Janík: RPC: do not print ping info in getpeerinfo when no ping received yet, fix help
< GitHub48>
bitcoin/master d97101e Wladimir J. van der Laan: Merge #7842: RPC: do not print minping time in getpeerinfo when no ping received yet...
< GitHub93>
[bitcoin] laanwj closed pull request #7842: RPC: do not print minping time in getpeerinfo when no ping received yet (master...20160408_getpeerinfo_no_ping_yet) https://github.com/bitcoin/bitcoin/pull/7842
< gmaxwell>
using decode raw transactions to determine the number of bytes in a hex string seems weird... also, was mentioned, it doesn't include signatures at the point where you're estimating the fee.
< gmaxwell>
when you're author a transaction the fee calculation should be based on how many and what kind of inputs you're using and how many and what kind of outputs.
< jonasschnelli>
Agree. decoderawtransaction(<hex>)['size'] is stupid
< jonasschnelli>
Only makes sense if you also need other data form the call.
< sipa>
gmaxwell: i guess it would be convenient past segwit
< sipa>
to get vsize
< cfields>
meeting?
< gmaxwell>
Hi all.
< wumpus>
I guess?
< sdaftuar>
hello
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu Apr 14 19:00:50 2016 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< btcdrak>
gavel wont be attending due to last week's beating.
< cfields>
heh
< gmaxwell>
"If I have to."
< wumpus>
topics?
< gmaxwell>
Whats up with the 0.12.1 release? I haven't been paying attention to it.
< morcos>
status of 12.1
< btcdrak>
0.12.1 status
< wumpus>
0.12.1 is at rc2
< wumpus>
haven't heard of any issues with it
< gmaxwell>
There was a 0.12.1(rc) block mined in the last week.
< sipa>
i suggest we go ahead with release?
< jonasschnelli>
+1
< btcdrak>
+10
< sipa>
-11
< btcdrak>
doh!
< gmaxwell>
+1i.
< sipa>
gmaxwell: interesting; block version 0x2..?
< sipa>
but not classic
< Luke-Jr>
we should release 0.12.1 when 0.12.1 is observed to be released.
< sipa>
Luke-Jr is the first member of the club containing Luke-Jr as first member
< gmaxwell>
sipa: yes, it seems we kinda flubbed the part of the motivation with the BIP9 starting dates to remove the possibility of warings in advance of a release. :)
< Luke-Jr>
that sounds lonely.
< wumpus>
usually we wait a week after the release of the binaries before labeling the rc as final, the binary release was only 3 days ago
< sipa>
wumpus: ok
< wumpus>
but if you insist
< sipa>
topic suggestion: segwit and backports
< btcdrak>
let's insist
< jonasschnelli>
Yes. We should wait at least one week.
< gmaxwell>
sipa: but at least thats just a one time event.
< gmaxwell>
well 1 week - 3 days is still before the next meeting. So the action should be release 0.12.1 prior to the next meeting. :)
< Luke-Jr>
could start the gitian building now and wait the rest of the week to publish?
< jonasschnelli>
the tag could be seen as "release"
< btcdrak>
1 business week is 5 days :)
< wumpus>
good to hear a block would mined with 0.12.1 though, hadn't heard that yet
< wumpus>
s/would/was
< btcdrak>
why not tag 0.12.1 we can build over the weekend and release on Monday?
< jonasschnelli>
Why not tag on monday and build on monday?
< jonasschnelli>
You don't need to handhold you bulder?
< sipa>
wumpus decides
< gmaxwell>
the building has latency.
< wumpus>
you're too eager
< btcdrak>
gotta build, cfields needs to sign etc.
< gmaxwell>
in the future we should figure out how to deseralize builds of the final release and publication of the tag.
< wumpus>
huh
< jonasschnelli>
we could agree on a commit and build that and tag later.
< gmaxwell>
the fact that we've been doing "releases" that get announced and circulated ahead of the project's announcement and then having binary days delayed is suboptimal.
< sipa>
we need to have gitian build inside a homomorphically encrypted environment, so that we can verify binary correctness, but only release the key at release time
< wumpus>
is this still serious?
< cfields>
fwiw, dropping the commit hash in the binary would create a few more options
< sipa>
wumpus: no
< wumpus>
ok...
< wumpus>
next topic?
< jonasschnelli>
[21:05:02] <sipa>topic suggestion: segwit and backports
< btcdrak>
segwit backports
< wumpus>
#topic segwit and backports
< sipa>
so, segwit branch is currently on top of 0.12.1
< btcdrak>
so you'd be forward porting it to master
< sipa>
with a set of backports from master and some PRs first
< Luke-Jr>
it would be ideal if the same branch can merge into both 0.12 and master, but that seems unlikely for segwit
< sipa>
i don't know what the best way forward is, but i think we're close to PRing it for bitcoin
< gmaxwell>
FWIW on the other implementations front, btcd has what looks like a more or less complete implementation of the consensus changes for segwit and is interoperating with segnet4.
< Luke-Jr>
gmaxwell: nice!
< cfields>
sipa: woohoo!
< wumpus>
gmaxwell: "and then having binary days delayed is suboptimal." why don't you ever gitian-build? :p
< sipa>
do we rebase on master, have that merged first, and then "magically" come up with a 0.12 backport? :D
< sipa>
which happens to include the necessary backports
< btcdrak>
sipa: I hear you are good at magic
< Luke-Jr>
sipa: what's the current segwit branch?
< Luke-Jr>
without segnet ideally
< sipa>
Luke-Jr: segwit4 and segwit (the first is where i continuously rebase, the second i push to whenever there is something significant)
< wumpus>
the reason for the delay is that a certian minimum threshold of number of people have to build it, submit signatures, etc., the only way to speed it up wouldb e if people build it quicker, although it's already lots faster than it used to be
< sipa>
Luke-Jr: they have segnet, though the commits for segnet are separate and can easily be skipped
< gmaxwell>
wumpus: if you'd like me to do so, I will. (re gitian builds)
< sipa>
that's anohter question: do we merge segnet?
< btcdrak>
gmaxwell: yes please.
< wumpus>
gmaxwell: I dont mind, but please don't complain if you're not involved okay :)
< btcdrak>
sipa: do we really need segnet?
< sipa>
btcdrak: i don't think so
< morcos>
This might sound crazy, but I'd be in favor of merging the segwit PR very quickly. I think we should make the PR's for master and 0.12 at roughly the same time. And then we should all bust our ass to review them at roughly the same time.
< sipa>
that sounds fine to me
< morcos>
If they are outstanding for a while, then we'll all be reviewing different code as its rebased or merge conflicts are addressed or whatever
< btcdrak>
morcos: no complaints
< morcos>
We should just rip the bandaid off and get it in
< gmaxwell>
wumpus: my complaint wasn't about any of the people, for sure. if it were, I'd just build. Also, it wasn't really a complaint with anything we're doing. It's a complaint that the community prereleases our releases, but we can't stop that; so the best fix is to reduce latency, I think.
< morcos>
And then every other PR out there can get conflicted once and be done with it
< btcdrak>
in all fairness there has been so much testing and peer review and help from downstream consumers with segwit.
< sipa>
ok, will PR soon
< jonasschnelli>
A quick merge sound good. Also, nobody can complain we haven gave any change for review because there is already an open pr since weeks/month.
< sipa>
both master and 0.12
< btcdrak>
super. jonasschnelli you should close your preview PR so it doesnt get confusing
< gmaxwell>
I've felt bad about working on 0.13 in areas that I know will need to be tweaked for segwit, so I can agree with that. As soon as 0.12 is tagged this could be done. If we end up needing a 0.12.2 without segwit we could branch 0.12 pre-segwit.
< jonasschnelli>
Yes. Will close. Its pointing also to the wrong branch.
< btcdrak>
gmaxwell: maybe we dont need to over complicate if the PR/backport will come soon.
< morcos>
My main point is that it is goign to be a lot of work to adequeately review and test segwit. It'll be more efficient use of everyones time if we concentrate that effort at the same time.
< sipa>
that makes sense
< sipa>
any other topics?
< wumpus>
what about c++11 status?
< sipa>
ack
< wumpus>
#topic c++11 status
< * sipa>
summons cfields
< cfields>
wumpus: travis has enabled caching but only for flagged projects. I mailed this morning and asked for the flag.
< sipa>
cfields: is that the final blocker?
< cfields>
sipa: as far as i know, yes
< sipa>
awesome
< wumpus>
that was already the case right? open source projects needed a flag to support caching. But we need a new one now?
< wumpus>
hopefully we can start supporting it this week then
< gmaxwell>
would we be using C++11 in 0.13 then?
< cfields>
if there's a delay with this step, I'm completely ok with saying "screw it" and disabling whatever we need so that we can limp along until it's not blocking anymore
< wumpus>
gmaxwell: that was the plan
< Luke-Jr>
Travis is not willing to flag just any projects. We should try to avoid relying on it.
< gmaxwell>
wumpus: good. :)
< cfields>
so, I've been hacking on c++11 on a branch of mine for a while. One thing that's clear is that we need a policy on what modernizations we'll allow...
< cfields>
otherwise, I'm afraid it'll be endless PRs
< cfields>
Luke-Jr: it's just a flag while it's in testing
< sipa>
cfields: there are a couple of mechanic translations i think we'll want ayt some point
< gmaxwell>
We should keep modernization PRs slow initially. Then do the mechanical updates (e.g. replace boost with C++11).. and only after go back and make more changes.
< gmaxwell>
at least thats my intuition.
< sipa>
cfields: BOOST_FOREACH and boost::thread are good examples
< wumpus>
replacing boost is far from mechanical
< wumpus>
well ok some parts
< cfields>
sipa: yes, for sure. what I worry about it thousands of PRs that sprinkle in std::move all over the place.
< morcos>
is there any downside to using c++11 in new code while not yet doing any modernization PR's?
< sipa>
but for example: adding move constructors instead of swaps everywhere
< gmaxwell>
in particular I think it's probably unwise to do many moderinization updates when non C++11 versions are still supported.
< Luke-Jr>
someone remind me why are we not doing a release that fails with C++11 at configure, before actually introducing C++11 code?
< cfields>
(and emplace, which would be significant in several places)
< BlueMatt>
Start with only new code is C++11, then only boost-replacement, then revisit
< BlueMatt>
?
< gmaxwell>
because that would vastly complicate backports.
< sipa>
one option is building with c++11 and c++03 both for a while
< morcos>
I would say we have a lot on our plate in the next couple months, and we should just say no modernaization right now. (sounds like what BlueMatt said)
< wumpus>
meh
< cfields>
morcos: i think that would be my preference
< wumpus>
we can already build with c++11 for quite a while, that's nothing new
< sipa>
ok
< sipa>
just not too actively replace things initially
< BlueMatt>
I mean for 0.13 Id say dont actively replace anything unless its a big performance win
< Luke-Jr>
at the very least, let's add something to configure that fails if C++11 is not supported?
< wumpus>
we've never let refactors through too quickly
< wumpus>
correctness is most imporant
< Luke-Jr>
that way we get user reports
< gmaxwell>
Matt has an upcoming PR that uses C++11 that it might be nice to not have to port to C++03.
< wumpus>
Luke-Jr: yes, see the pull I quoted
< wumpus>
Luke-Jr: it fails without c++11 support
< BlueMatt>
gmaxwell: for the tcp stuff I think it actually doesnt matter, I was just lazy...udp would be annoying to fix, I think
< cfields>
wumpus: let's use a real example.. adding move ctors to our containers
< cfields>
yea or nay?
< sipa>
cfields: yay, please... but maybe not immediately
< BlueMatt>
cfields: for 0.13, I'd vote only new code, maybe boost replacement, too
< wumpus>
cfields: if it helps performance, absolute yay
< sipa>
(prevector specifically is unsafe to use for more complex types now)
< BlueMatt>
but I'm always the annoyingly conservative one, sooooo
< wumpus>
boost replacement is too late for 0.13
< sipa>
wumpus: agree
< wumpus>
too much work
< BlueMatt>
wumpus: ehh, I meant partial-boost-replacement
< wumpus>
we can do some minor things and use it in new code, but aiming to replace boost enitrely just won't work
< gmaxwell>
New code +1, especially new features, but as I mentioned, I think we should avoid making backports to 0.12 harder while 0.12 is still supported.
< sipa>
just turning on c++11 may already give a performance improvement, because STL magically gets move constructors
< jonasschnelli>
At least we could throw out boost::filesystem soon (there is no c++11 equivalent).
< wumpus>
to me it seems we want to backport so much to 0.12 it is starting to make more sense to do 0.13 sooner
< wumpus>
do a release from master
< wumpus>
may work better with cfields' holiday too
< morcos>
wumpus: but then not release segwit for 0.12?
< cfields>
stupid inconvenient honeymoon...
< morcos>
thats the issue right.. are we only going to support it on 0.13?
< sipa>
cfields: priorities!
< jonasschnelli>
heh!
< gmaxwell>
morcos: I don't think we should do that.
< sipa>
i think we should backport segwit to 0.12
< wumpus>
(we have to change the release schedule a bit anyway)
< sipa>
let's not push users too much
< btcdrak>
if we release 0.13 we still have to backport to 0.12
< btcdrak>
since we support this and prev version
< gmaxwell>
wumpus: what kind of schedule change are you thinking of?
< gmaxwell>
Can we talk about what notable features are still in flight that people are working on with an intent of targeting 0.13?
< btcdrak>
maybe backport to 0.12 and release 0.13 early.
< morcos>
whats the hurry to release 0.13 early in that case?
< phantomcircuit>
jonasschnelli: was that benchmark of leveldb vs lmdb on a system with a hdd or ssd?
< jonasschnelli>
ssd
< phantomcircuit>
interesting
< jonasschnelli>
(can be discussed after the meeting)
< gmaxwell>
btcdrak: there are things people are working on that aren't PRs yet.
< Luke-Jr>
gmaxwell: I'd like to rework the config/arg stuff, but I don't know I'll have time to finish it before 0.13 forks
< BlueMatt>
wumpus: next topic? we didnt really come to a conclusion at all here yet? what is the release schedule for 0.13 looking like, and does that mesh with turning on C++11 and allowing new code to use it?
< Luke-Jr>
(and there's no reason it can't wait for 0.14 etc)
< wumpus>
well the change we have to make in the 0.13 release schedule is to clear june - this works btter for cfields
< wumpus>
but we can just as well postpone it to later, moving it ealier was just a stupid idea
< cfields>
no reason to change just for me?
< btcdrak>
oh just a note regarding ctaes, independent review is in progress from one of Matthew Green's graduates, hopefully available in a couple of weeks.
< wumpus>
well the rc cycle was exactly planned that
< wumpus>
then*
< gmaxwell>
wumpus: I dunno that its stupid. But it should be made with consideration.
< wumpus>
can do that in july instead of june , no problem
< morcos>
gmaxwell: yes, i'm working on 2 things that might be nice to get in. an improvement to fee estimation and some measurement of policy alignment. they are goign to be annoying for anyone to review, but they also stand alone.
< gmaxwell>
BlueMatt: C++11 is getting turned on, in a week barring emergencies if I understood above. And it sounded like new things could use it, but we'd avoid going and refactoring for more until 0.14.
< morcos>
wumpus: yes i like the idea of pushing back 0.13 a bit
< sipa>
gmaxwell: ack
< BlueMatt>
gmaxwell: ACK
< jonasschnelli>
Features I'd like to see in 0.13: wallet/RBF (+GUI), simple profiles (maybe GUI only), BIP32 (probably not in 0.13 because of lack of review)
< sdaftuar>
regarding 0.13, i thought it might make sense to push the freature freeze back slightly, since there's a meetup happening in zurich a week afterward that many of us would be at
< BlueMatt>
wumpus: ok, so release schedule stays as-is for now?
< wumpus>
BlueMatt: I proposed moving the rc phase to july instead of june
< jonasschnelli>
sdaftuar: good point.
< Luke-Jr>
in talking about using C++11, we won't need GCC 5, right?
< Luke-Jr>
because GCC 5 is not really a reasonable minimum yet
< BlueMatt>
when does cfields gete back?
< morcos>
wumpus: ack
< cfields>
Luke-Jr: 4.8 should be fine, i think
< wumpus>
no, we don't need gcc5, the features we should use of c++11 shoudl work in gcc4.8+
< Luke-Jr>
k
< sipa>
Luke-Jr: i believe (but cfields correct me) that the features from c++11 we'd want are almost entirely support in 4.8
< wumpus>
otherwise we can't use trusty for building
< cfields>
BlueMatt: july4ish
< wumpus>
and that's the whole point
< sipa>
that's clear, thank you
< cfields>
BlueMatt: if it turns out to be too problematic, i can revisit the dates.
< gmaxwell>
Luke-Jr: C++11 is fully supported in 4.9, and 4.8 has basically everything. I think we could reference 4.8 for compatibility.
< BlueMatt>
cfields: lol, dont change honeymoon for us
< sipa>
all known ABI issues result in link errors
< cfields>
for that matter, gcc6 will build with c++14 by default. So either way (maybe even for 0.12 backport), we need to specify the standard we're using.
< wumpus>
hah, c++14
< gmaxwell>
As far as other in-flight stuff, Matt has implemented efficient block relay; related to a design I've been circulating for a long while. He has code up, and it works pretty well.. I get about a 96% reduction in block bandwidth. The protocol needs a few tweaks still but once in, it should be able to send the vast majority of blocks in 0.5 round-trip times (plus whatever awful overhead TCP adds),
< wumpus>
not looking forward to going through this again :(
< gmaxwell>
the rest will need 1.5 RTT. I've started on a BIP. Matt has also been working on some other things that go further and eliminate even more latency, though that work is likely only going to be interesting to miners.
< BlueMatt>
I think gmaxwell is more excited about tcp-smaller-block relay since his internet at home sucks
< gmaxwell>
I think with segwit upcoming I'd really like to see that work make its way into 0.13, since we really need propagation time mitigations, and the relay network client only goes so far.
< cfields>
nice
< cfields>
BlueMatt: you have a branch somewhere?
< BlueMatt>
I see it more as foundational work that is useful for compression on the wire, but is primarily for udp-based relay which is nice and fast :)
< wumpus>
cfields: how does it make sense that they will build with c++14 by default?!
< BlueMatt>
tcp stuff is close to ready with one or two stubs to be replaced, pr next week when I'm off vacation, udp stuff is a bit longer-term
< cfields>
wumpus: i suppose because it should all be forward compat, in theory
< sipa>
wumpus: i think c++14 has less impact on standard libraries
< cfields>
BlueMatt: thanks
< sipa>
and is much more incremental than c++11 was
< BlueMatt>
cfields: if you want to tackle the udp-wip changing to libevent I'd owe you :)
< cfields>
yes, c++14 as i see it is kinda a c++11 fixup
< wumpus>
ok...
< Luke-Jr>
cfields: IIRC, binaries (incl libraries) compiled with C++11 mode are incomptible with libraries compiled with C++98 mode.
< sipa>
c++14 finally has a solution for the exponentially-sized error messages
< Luke-Jr>
sipa: ok
< cfields>
Luke-Jr: that has nothing to do with how we compile though. That'll be the case either way.
< gmaxwell>
BlueMatt: mostly trying to avoid bleeding out over load increases permitted by segwit. :)
< wumpus>
sipa: nice!
< cfields>
BlueMatt: heh yes, we should sync up some
< Luke-Jr>
cfields: ? our dependencies are compiled with C++98 in almost all cases today
< wumpus>
anyhow any other meeting topics?
< kanzure>
MAST bip status?
< gmaxwell>
lol. way too premature for discussion here.
< btcdrak>
kanzure: it got published BIP114
< cfields>
Luke-Jr: yes, and they'll be switched to c++11 for the ones we build. Otherwise it's a cointoss what users have on their systems, sticking with c++03 could be equally incompatible for them.
< wumpus>
Luke-Jr: cfields let's leave the small implementation details about the c++11 switch to after the meeting
< cfields>
ack
< Luke-Jr>
cfields: we do not build our dependencies typically.
< Luke-Jr>
possible topic?: using SHA256d for segwit
< wumpus>
sha256d?
< sipa>
Luke-Jr: i'd rather not break all tests and downstream code that was written yet... though we'll want new segwit script versions soon, and they can use sha256d hashing
< sipa>
wumpus: background: p2wsh scriptPubKeys contain a SHA256 (not double SHA256) of the witness program
< paveljanik>
double sha256
< sipa>
which is incompatible with the "P2SH^2" scheme that greg proposed a while ago
< wumpus>
oh right
< wumpus>
it would be consistent
< gmaxwell>
The only advantage of it that I'm aware of is the p2sh^2 trick which would need new address types and such to ever get used. Disadvantage is that its slow.
< gmaxwell>
And consistency, I suppose.
< sipa>
... which interacts with the discussion of addresses for native segwit
< Luke-Jr>
gmaxwell: there's no address types for bare segwit yet
< sipa>
Luke-Jr: i do plan on proposing one
< sipa>
(though not immediately)
< Luke-Jr>
yes, my point is that we have an opportunity to avoid breaking compatibility
< gmaxwell>
we'd purposefully moved them out to avoid shedpainting and address-type-deployment issues from harming segwit consensus code deployment.
< phantomcircuit>
gmaxwell: sha256d also prevents length extention attacks although i dont see how that's applicable here at all
< gmaxwell>
Luke-Jr: I think that nativesegwit of that initial type is not likely to see long term use, see also that MAST bip.
< Luke-Jr>
sipa: am I incorrect in assuming adjusting downstream code for this would be super trivial?
< sipa>
Luke-Jr: probably, but we also don't have a way to deploy P2SH^2 easily
< Luke-Jr>
gmaxwell: yes, but I think sipa wants an address format that works for all segwit outputs, regardless of version
< Luke-Jr>
sipa: that's not something we need to bundle with making it possible
< gmaxwell>
we should close the meeting and continue the discussion.
< sipa>
ok
< gmaxwell>
no resolution to this will happen this instant.
< sipa>
#shutdown -h now meeting
< jonasschnelli>
sudo!
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Apr 14 19:59:08 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< paveljanik>
jonasschnelli, no need for sudo once you have # ;-)
< jonasschnelli>
nerds! oO
< jtimon>
meeting?
< gmaxwell>
So, wrt p2sh^2. That got proposed eons ago and I think it's unlikely to ever get implemented.
< jonasschnelli>
jtimon: hahaha
< gmaxwell>
jtimon: an hour ago.
< jtimon>
oh...
< phantomcircuit>
timezones strike again
< jtimon>
well, read the logs I guess
< btcdrak>
jtimons needs to run an ntp daemon
< kanzure>
this is why we should abolish time zones
< Luke-Jr>
gmaxwell: isn't the only major hold-up the backward compat issue?
< Luke-Jr>
kanzure: tonal time! :D
< kanzure>
i would be willing to accept tonal time if it means no more time zones
< paveljanik>
meeting bot should be modified to ping people beforehand :-)
< jonasschnelli>
jtimon: I can add a cron job that sends you a PM 10mins before the meeting? But people where using alarm-clock back in the "old days". :)
< gmaxwell>
Luke-Jr: no, well it requires yet another kind of witness data in transactions.
< sipa>
Luke-Jr: plus that it's easily bypassable, and we could end up encouraging people to send to miners directly
< jtimon>
phantomcircuit: I would share last week it was a 22:00, but it may may that I forgot about my last mistake
< Luke-Jr>
gmaxwell: that's pretty simple now with segwit
< gmaxwell>
the pairing crypto based scheme I came up with didn't have those issues but it's slow.
< jtimon>
jonasschnelli: that would be indeed useful
< Luke-Jr>
sipa: that assumes miners are willing to cooperate in the spam
< Luke-Jr>
actively
< sipa>
Luke-Jr: why wouldn't they, if people want to pay
< Luke-Jr>
at the very least, such miners would likely opt to charge higher fees (I hope)
< jtimon>
never mind much, I will eventually adapt to this without putting it in google calendar, you'll see
< sipa>
p2sh^2 is also incompatible with forward compatible address types
< gmaxwell>
sipa: I dunno, thats kind of an argument against any non-stanardness. Dust limits have been effective.
< sipa>
gmaxwell: fair enough
< Luke-Jr>
not necessarily? if all future address types are p2sh^2
< gmaxwell>
But yes, it would be incompatible with address types that change the top hashing scheme.
< wumpus>
jtimon: date -u , if it shows thursday 19:00-20:00 it's time for the meeting :)
< sipa>
Luke-Jr: i guess if we stick to 1) there is some scheme below that results in 32-byte data 2) on top of that there is single SHA256
< sipa>
then it can be compatible
< Luke-Jr>
we tried a one-size-fits-all address type for p2sh, and that failed with segwit. I'm not sure it's possible to make future-proof here really.
< gmaxwell>
No but we can be a bit more future robust than p2sh was.
< gmaxwell>
There is a greater archectural question around the pace of script changes.
< gmaxwell>
If there are a lot of little script changes that do nothing other than add OP_SWEETANDSOURCHECKSIG then its essential there be an address type that automagically covers them.
< CodeShark>
how is simply encoding the scriptPubKey not fully general?
< sipa>
CodeShark: it doesn't cover P2SH^2
< Luke-Jr>
gmaxwell: hopefully all scripts will be inside the witness
< sipa>
CodeShark: but apary from that, yes
< sipa>
Luke-Jr: the witness version number needs to be outside
< Luke-Jr>
sipa: yes, but that doesn't interfere with p2sh^2
< sipa>
CodeShark: you have an address that encodes SHA256(script), and the output contains SHA256(SHA256(script))... but for relay you're required to also provide SHA256(script) (which does not go into blocks, however)
< instagibbs>
can we get the post on p2sh^2? I read it like 3 years ago, forgot the details
< instagibbs>
oh ok
< sipa>
CodeShark: this guarantees that the data in the output is a hash, and not data storage
< gmaxwell>
CodeShark: and also makes it somewhat harder to just send funds to random "addresses" culled from the blockchain (you need the preimage side information).
< gmaxwell>
A practice which is a major accounting nusance for many parties.
< sipa>
b.i will of course publish all the preimages they see in the network
< gmaxwell>
CodeShark: not in the UTXO set, of course that stuff could still end up in witnesses.
< instagibbs>
sipa, sigh, you are of course correct
< instagibbs>
or just captured by sybil anti-laundering services
< gmaxwell>
sipa: I wouldn't be so sure.
< sipa>
there is also a minor downside to using sha256^2 for script hashes, namely that it means wallet needs to maintain two indexes for scripts (one for p2sh, one for p2wsh, because their hashes are not compatible)
< sipa>
very minor, and possibly something that can't be maintained for future witness script version anyway
< Luke-Jr>
nah, addresses are either p2sh or p2wsh, not both.
< sipa>
Luke-Jr: scripts, not addresses
< sipa>
but i agree it's minor
< wumpus>
* [new tag] v0.12.1 -> v0.12.1
< Luke-Jr>
I don't follow. If it's used in a p2sh address, you don't need to index the script for p2wsh..?
< sipa>
Luke-Jr: you have an index from Hash160 -> script for known scripts
< sipa>
Luke-Jr: that can be used for matching both P2SH and P2WSH
< Luke-Jr>
but you don't need scripts for p2wsh in that index
< sipa>
perhaps it's better that it's a separate index
< sipa>
maybe
< Luke-Jr>
oh, you mean just having two different indexes at all, even if non-overlapping
< sipa>
well you need an index anyway for p2wsh scripts
< sipa>
the question is whether it can be the same one or not :)
< * sipa>
afk
< CodeShark>
we could in principle add another marker to the address to indicate some operation to be performed on the data to transform it into the scriptPubKey
< CodeShark>
this would give us flexibility to apply other hash functions or transformations later on
< CodeShark>
but I would like to avoid special cases