< dcousens>
I know ZMQ doesn't guarantee your message, but I'm currently getting a ~1/3 miss rate for hashblock messages, could hashtx messages just be flooding it too much?
< gmaxwell>
dcousens: you could try commenting that out and see if it solves it?
< dcousens>
gmaxwell: ? would you rather I make an issue? Its a genuine question
< gmaxwell>
I don't have anything useful to suggest other than from my totally ignorant perspective the idea that the flood of tx is contributing to loss sounds plausable to me.
< jcorgan>
dcousens: there could be a number of issues causing that. for one, zmq PUB ports are designed to drop msgs to slow listeners, but i don't recall offhand what the boundary conditions are
< jcorgan>
if you can document it in more detail as an issue, i can take a look at it
< dcousens>
jcorgan: it appears no ZMQ sequence numbers are being skipped - so I'm going to debug and determine if the issue lies elsewhere in bitcoind or otherwise
< dcousens>
jcorgan: nevermind, it appears when a new block comes in with transactions unseen, the spam causes mass message loss including the hashblock itself
< dcousens>
I get 2-3000 sequence number differences
< jcorgan>
eww
< jcorgan>
you can separate those onto two different ports
< dcousens>
jcorgan: will check that out, but cheers for help :)
< bitcoin-git>
[bitcoin] sdaftuar opened pull request #10006: [0.14 backport] Don't require segwit in getblocktemplate for segwit signalling or mining (0.14...backport-9955) https://github.com/bitcoin/bitcoin/pull/10006
< gmaxwell>
what time is the meeting? my calandar shows unmoved but the US recently went through DST so I think my calandar is wrong.
< gmaxwell>
though tne entry is set to iceland time.
< achow101>
gmaxwell: the meeting is 1 hour after whatever the meeting time was for you last week
< gmaxwell>
what is the scheduled time?
< achow101>
7 pm utc
< gmaxwell>
ok good.
< Lightsword>
what do I have to do to mine directly to a witness address on testnet? I tried setting it to 2NF1bac2Q8CJem1sy95pcZ82kfaMaPBuRhm but the generation transaction address in the block was n3HKtNgumrvBFB2c4RWXLTttAhTPxWPSQo
< bitcoin-git>
[bitcoin] NicolasDorier opened pull request #10007: [QT] Remove SendToSelf, and break down its payouts (master...watchonlylabel2) https://github.com/bitcoin/bitcoin/pull/10007
< bitcoin-git>
[bitcoin] practicalswift opened pull request #10008: [trivial] Fix a typo (introduced two days ago) in the default fee warning (master...fix-typo-in-default-fee-warning) https://github.com/bitcoin/bitcoin/pull/10008
< bitcoin-git>
bitcoin/master a3ca43b practicalswift: [trivial] Fix a typo (introduced two days ago) in the default fee warning
< bitcoin-git>
bitcoin/master d42729a Jonas Schnelli: Merge #10008: [trivial] Fix a typo (introduced two days ago) in the default fee warning...
< bitcoin-git>
[bitcoin] jonasschnelli closed pull request #10008: [trivial] Fix a typo (introduced two days ago) in the default fee warning (master...fix-typo-in-default-fee-warning) https://github.com/bitcoin/bitcoin/pull/10008
< MarcoFalke>
I don't think they meant the wallet specifically
< wumpus>
ah, okay
< luke-jr>
NicolasDorier: is your PR description outdated then? O.o
< NicolasDorier>
no
< NicolasDorier>
what I do is
< NicolasDorier>
in the case where
< NicolasDorier>
All input and all output are from me
< NicolasDorier>
show 1 lines for the debit. Which Label is taken from the first input
< luke-jr>
each output should get one "send" and one "receive" with the same label based on the output address
< luke-jr>
inputs don't have labels
< NicolasDorier>
I take the ScriptPubKey of the first input and use that as the label for the debit
< NicolasDorier>
then 1 line per output
< luke-jr>
NicolasDorier: don't do that.
< NicolasDorier>
the idea is that when you read the transaction log, you can follow where the coin leave and where it go. This is very useful for every 2nd layer protocols because bitcoin core can show that is happening
< NicolasDorier>
ok so let's take an example with Lightning
< NicolasDorier>
Imagine that you fund a channel
< NicolasDorier>
then later you close the channel
< NicolasDorier>
what you want to see in the log is
< luke-jr>
the GUI isn't 2nd layers or Lightning.
< NicolasDorier>
(na) [+1.0]
< NicolasDorier>
Channel [-1.0]
< NicolasDorier>
Channel [+1.0]
< NicolasDorier>
(na) [-1.0]
< NicolasDorier>
oops
< NicolasDorier>
I mean
< NicolasDorier>
no that is correct sorry
< NicolasDorier>
SendToSelf are only relevant for above layer protocols
< NicolasDorier>
and this is a way to make the SendToSelf actually usefull to anything
< NicolasDorier>
on the UI
< luke-jr>
wallets used to watch L2 stuff simply shouldn't be used directly by end users
< NicolasDorier>
this is what joinmarket is doing
< NicolasDorier>
and there is good reason imho
< wumpus>
well we can try to accomodate for it, but not if it breaks other valid use cases
< NicolasDorier>
coin tracking is hard.
< NicolasDorier>
coin selection is hard
< NicolasDorier>
I do not want to code this stuff myself
< NicolasDorier>
I will mess it up
< wumpus>
I think we should try to help projects like joinmarket and lightning where possible
< bitcoin-git>
bitcoin/master c459d50 Wladimir J. van der Laan: build: Probe MSG_DONTWAIT in the same way as MSG_NOSIGNAL...
< bitcoin-git>
bitcoin/master 25da1ee Wladimir J. van der Laan: build: cleanup: define MSG_DONTWAIT/MSG_NO_SIGNAL locally...
< bitcoin-git>
bitcoin/master a4d1c9f Wladimir J. van der Laan: compat: use `unsigned int` instead of `u_int`...
< bitcoin-git>
[bitcoin] laanwj closed pull request #9921: build: Probe MSG_DONTWAIT in the same way as MSG_NOSIGNAL (master...2017_03_cloudabi_netcompat) https://github.com/bitcoin/bitcoin/pull/9921
< luke-jr>
wumpus: I guess his goal is to display L2 semantics in the GUI without the GUI knowing L2 stuff. Not sure.
< MarcoFalke>
maybe a good time to clear the release notes on the 0.14 branch
< NicolasDorier>
wumpus: The goal is to make it possible to follow the coins from L2 transfers from the Bitcoin qt interface
< NicolasDorier>
for lightning
< NicolasDorier>
for example
< NicolasDorier>
if the channel get open then closed
< NicolasDorier>
you would like to see
< wumpus>
NicolasDorier: aren't we missing the metadata for that?
< wumpus>
MarcoFalke: yes, good idea
< NicolasDorier>
"Channel X" +1 BTC
< NicolasDorier>
"Channel X" -1 BTC
< NicolasDorier>
labels being submitted by the Layer 2 process
< wumpus>
ok, sounds fair enough to me
< NicolasDorier>
in the PR I gave an example with tumble bit where the coins flow internally through 3 different transactions
< NicolasDorier>
and I would like users to see something like
< NicolasDorier>
Tumbler [1.0]
< NicolasDorier>
Offer [-1.0]
< NicolasDorier>
Offer [1.0]
< NicolasDorier>
Escrow [-1.0]
< NicolasDorier>
Escrow [1.0]
< NicolasDorier>
(n/a) [-1.0]
< NicolasDorier>
so you can see your money go to the escrow, to the offer, then ultimately to the tumbler
< wumpus>
I guess tumbler, offer and escrow are separate pools of coins which shouldn't mix? how is this enforced?
< NicolasDorier>
not pools of coins
< wumpus>
when they're all in one wallet
< NicolasDorier>
there is no pool of coins that should not mix
< wumpus>
what if the user does a spend from bitcoin core?
< NicolasDorier>
he can. The way tumbler bit work now is to drain the bitcoin core wallet until empty to the tumbler
< wumpus>
the coin selection algorithm can pick from all of those utxos, despite being allocated to something (as "send to self") by L2 software
< NicolasDorier>
for coin separation I want the multi wallet
< wumpus>
right.
< NicolasDorier>
but right now yes, the user share the same pool of coins
< NicolasDorier>
the idea is that he receive money to the bitcoin core and tumblebit passively send it through the tumbler
< NicolasDorier>
if he does not want all the coins drained then he need separate wallet
< NicolasDorier>
But basically I am labelling the escrow, offer, and tumbler addresses. So that the user can see that the money does not get lost
< wumpus>
yes it makes sense to have some way of displaying that in the wallet GUI
< wumpus>
doesn't it need more metadata than just send-to-self though?
< wumpus>
I guess it needs some way to mark transactions, for example through RPC?
< NicolasDorier>
you do not mark transactions
< NicolasDorier>
you mark addresses
< NicolasDorier>
wumpus: the layer 2 instruct which address to watchonly
< NicolasDorier>
with their label
< NicolasDorier>
despite the UI show label as one per transaction, in reality it is one per addresse
< wumpus>
ok, metadata for the address then
< NicolasDorier>
on my side I do not need more metadata to show correctly
< NicolasDorier>
the PR I have done show quite well the flow of money through the different stages
< wumpus>
but it may interfere with other uses of send-to-self which should not be shown like this
< NicolasDorier>
a proposition that luke-jr would be fine with I think is if I do not add one entry for debits so instead of
< NicolasDorier>
Tumbler [1.0]
< NicolasDorier>
8:19 PM Offer [-1.0]
< NicolasDorier>
8:19 PM Offer [1.0]
< NicolasDorier>
8:19 PM Escrow [-1.0]
< NicolasDorier>
8:19 PM Escrow [1.0]
< NicolasDorier>
8:19 PM (n/a) [-1.0]
< NicolasDorier>
it would be
< wumpus>
in that case you'd need some special flag in the address metadata, instead of treating all of them the same
< wumpus>
okay
< NicolasDorier>
Tumbler [1.0]
< NicolasDorier>
8:19 PM Offer [1.0]
< NicolasDorier>
8:19 PM Escrow [1.0]
< NicolasDorier>
but I think it is not very clear
< NicolasDorier>
but indeed
< NicolasDorier>
for the debit entries, I just use the address of the first input
< NicolasDorier>
which might not work for all cases
< luke-jr>
NicolasDorier: the send+receive should always use the same label, with the current wallet structure
< NicolasDorier>
not sure what you nmean here
< wumpus>
luke-jr: but if it is *to self* that's kind of hard to define
< luke-jr>
wumpus: if the goal is to eliminate "send to self", it leaves just a send and a receive both with the same address
< NicolasDorier>
luke-jr: so how would you show ?
< NicolasDorier>
coming back soon
< luke-jr>
NicolasDorier: for each output, 1 send, and 1 receive, both with the same label/address
< NicolasDorier>
luke-jr: I think it is ugly and does not really help user to understand where his money go
< jonasschnelli>
Would be great if we could merge #9294
< bitcoin-git>
[bitcoin] TheBlueMatt opened pull request #10013: Fix shutdown hang with >= 8 -addnodes set (0.14 backport) (0.14...2017-03-exit-with-addnode-13) https://github.com/bitcoin/bitcoin/pull/10013
< cfields>
wumpus: grr, sorry for missing that in review
< wumpus>
cfields: hah yes we should have caught that one
< wumpus>
bah we're swamped in Wshadow pulls, exactly what we feared has happened, about every PR is followed by one that 'fixes' its shadow warnings
< wumpus>
until a sneaky bug gets introduced in an oversight while renaming variables
< ryanofsky>
probably been discussed before, but could this problem be avoided by enabling -Werror in travis?
< wumpus>
the problem is that different compilers have different perceptions of shadowing
< wumpus>
there is always someone with a compiler that shows more shadowing warnings than you
< wumpus>
clang shows fairly few, gcc 5 shows more, some gcc 4.x versoins go all out crazy
< ryanofsky>
oh, ok. still seems like it could be useful as mitigation, but maybe there are other reasons not to do it
< wumpus>
the reason not to do it would be that it causes a lot of extra overhead and irritation
< wumpus>
you say it is useful as mitigation, but are you helping review -Wshadow pulls? making sure they don't introduce bugs?
< wumpus>
anything that introduces so much ancillary diff noise is a risk in itself, what if something gets renamed wrongly. It was a bad idea to enable it by default.
< ryanofsky>
i have not, but would be happy to review and add some acks
< wumpus>
we hardly have enough reviewers for the pulls that actually add features or fix serious bugs
< ryanofsky>
oh i agree it's a dumb warning. sorry, i don't know whatever discussion happened previously around this issue
< cfields>
wumpus: i agree. It seemed like something that would eventually settle down and we could stick -Werror on it, but it's turned out to be very different in practice :(. It's not worth the pain it's causing
< ryanofsky>
was just suggesting the travis change as a way of shifting the burden of dealing with the warning from maintainers to contributors
< cfields>
ryanofsky: the discussion was triggered by a bug a long time ago that this would've caught, but the signal/noise is just too bad for it to be helpful
< gmaxwell>
What happened to the suggestion I made of tuning -Wshadow with the arguments?
< gmaxwell>
not successful and restricting it to things that compilers are consistent about?
< wumpus>
ryanofsky: the problem is that you'd have to run it against N different compiers in travis then, which would be good in itself, but another problem is that the turnaround cycle of travis is pretty slow - so you get one bucketload of warnings, fix it, travis complains about another. But sure, it'd certainly be much better to catch them before they're merged.
< cfields>
gmaxwell: from what i saw, the options available end up hiding the only useful warning
< cfields>
ryanofsky: for context, this started the discussion: #8102
< wumpus>
cfields: we didn't know better at the time
< wumpus>
I mean for C it would be clearly defined. It's just with C++ and all its nested scopes and global namespace symbols that compilers start to diverge on it
< gmaxwell>
cfields: ah, I thought one of them warned only about shadowing in the same function. Which sounded basically like the primary place where shadowing is possible in C, and still sometimes finds some bugs.
< wumpus>
secp256k1 also is smaller, and has a lot fewer dependencies, that helps with having not too much cruft in namespaces that can overlap. Though C/C++ probably makes the biggest difference.
< gmaxwell>
(of course, I've similarly not had problems in other C librarys... I really think it's just C vs C++ that is an issue here.)
< cfields>
gmaxwell: i suppose that doesn't hurt. Though, I was more interested in catching accidental global shadowing. Those are really rough to catch in review.
< cfields>
(ofc those are the ones that cause all the trouble here)
< gmaxwell>
It's still surprising to me that the C++ compilers can't be reliable in this, how the hell do they resolve the symbols if they can't find them. :P
< wumpus>
hehe I'm sure they can find them, it's just that they have different concepts of what is reported as a shadowing warning. May be something of a compiler vendor opinion as well, you can both blame clang for underreporting and gcc for overreporting.
< wumpus>
(FYI bitcoin core master compiles without WShadow warnings with clang 4.0 pre-something apart from one in db.cpp)
< gmaxwell>
I also worry about this constant fix stream causing us to introduce real bugs. :( and also undermining the utility of the warning.
< cfields>
jnewbery: ping
< jnewbery>
cfields: pong
< cfields>
jnewbery: got a sec to talk about the test movement?
< jnewbery>
sure
< bitcoin-git>
[bitcoin] jnewbery opened pull request #10015: Wallet should reject long chains by default (master...walletrejectlongchains) https://github.com/bitcoin/bitcoin/pull/10015
< Chris_Stewart_5>
meeting? Or are my time zones off?
< achow101>
meeting
< wumpus>
#meetingstart
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu Mar 16 19:01:26 2017 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< morcos>
i felt like we discussed that ad nauseum the first time around
< * instagibbs>
wimpers
< instagibbs>
yes
< sipa>
i don't remmeber the reason for it not being default
< wumpus>
yes I remember we already had long discussions about that
< jnewbery>
#9262
< instagibbs>
The idea being that now that we actually rebroadcast normally, and return txid, it wasn't required in general.
< gribble>
https://github.com/bitcoin/bitcoin/issues/9262 | Prefer coins that have fewer ancestors, sanity check txn before ATMP by instagibbs · Pull Request #9262 · bitcoin/bitcoin · GitHub
< morcos>
sipa: the reasoning is its a very reasonable use case that you'd just want the tx to go out as soon as some of its parents get confirmed..
< morcos>
it seems somewhat likely that it would be tricky to have anything smarter than that happen manually anyway
< sipa>
but the resulting behaviour seems very unexpected to users
< gmaxwell>
I don't see the reason for rejecting. Seems like a useless loss of functionality in most cases.
< morcos>
but the solution to that could be better informing them of the new behaviour
< jnewbery>
I can understand the use case, but user experience is terrible (hence already two issues opened by different users)
< gmaxwell>
What does it matter to you if your transaction 20 steps deep hasn't actually been announced yet? it will be announced when it can.
< sipa>
many reports of people who see their balance going down
< sipa>
and get scared
< gmaxwell>
Their balance is going down.
< morcos>
hmm...
< sipa>
gmaxwell: it is, but don't you think it's better to reject by default, so they know why it is going down?
< morcos>
gmaxwell: i assume he means they have a 10 BTC input, they spend 0.1 BTC and their balance goes down by 10
< sipa>
so they can re-enable it when they understand the effect
< morcos>
b/c the change isn't in mempool so it doesn't count towards balance
< gmaxwell>
morcos: okay now that is a bad effect, I didn't reaize it was doing that.
< gmaxwell>
realize*
< jnewbery>
see #10004 for good description of what the user sees
< gribble>
https://github.com/bitcoin/bitcoin/issues/10004 | After max chain of unconfirmed change transactions, last tx is missing from memory until rescan · Issue #10004 · bitcoin/bitcoin · GitHub
< gmaxwell>
jnewbery: or #9752 for the alternative
< morcos>
i briefly recall discussing that but i agree its bad so don't know why we just left it that way.. maybe b/c its not easy to do anything smarter?
< gmaxwell>
The balance being goofy is an issue, but I think that should be considered a seperate issue.
< gmaxwell>
I agree it shouldn't be left with the balance doing inexplicable things.
< sipa>
gmaxwell: you think we should include txn crediting the wallet that are not in the mempool?
< sipa>
gmaxwell: that would bring back all malleability craziness
< gmaxwell>
sipa: if it's the users own output? I think so.
< gmaxwell>
(and it's not conflicted.)
< morcos>
perhaps there needs to be a new category of pending txs
< sipa>
the conflict can be outside of the mempool
< sipa>
s/mempool/wallet/
< morcos>
The pending balance can include both the debit and the credit
< morcos>
But could get complicated
< jonasschnelli>
I tend to like this approach.
< sipa>
which approach?
< jonasschnelli>
pending txs cat
< sipa>
morcos: it's very hard to not double count things in the pending balance if they're spending from malleated versions of the same transaction
< gmaxwell>
I am dubious that your own mempool is actually that strong a protection here.
< jnewbery>
my view: simplest experience is best. Default should be to reject too-long-chain transaction from wallet and mempool. If the user wants to have long chains in wallet, that's fine but:
< jnewbery>
- it should be behind an explicit option
< jnewbery>
- user should understand that it could have unexpected impacts on things like getbalance()
< gmaxwell>
Transactions simply failing to create due to inexplicable internal things that the user does not understand and cannot easily understand is not a good expirence at all.
< morcos>
well look, this thing is an option, so its kind of ridiculous to spend this much time discussing the default. The solution no matter whether we change the default or not is more announcing of the effects in either case
< gmaxwell>
We were already getting complaints about inexplicable failures before.
< gmaxwell>
Many people do not have adequate error handling to deal with a sendtoaddress failing when the balance was sufficient.
< morcos>
But too long a chain, try again later is explicable
< sipa>
gmaxwell: i think seeing your balance going down inexplicably is worse than inexplicably failing to create a transaction (at least there can be an explanation message)
< gmaxwell>
I agree the balance is screwed up. But _that_ is the issue, not the rest.
< jnewbery>
take this discussion offline? I'm happy to receive feedback in #10015
< instagibbs>
Either way we can buff up the error messages to be far less scary, especially in this case.
< gmaxwell>
sipa: how about we remove all ability to sends funds entirely, then there never will be balance confusion?
< sipa>
gmaxwell: come on
< morcos>
well i wasn't implying we shouldn't discuss here, its kind of hard to have this discussion on a PR
< morcos>
gmaxwell: the balance issue is not easy to solve
< sipa>
sure, if balance was redefined completely we may be able to avoid that issue, but i don't even know where to start
< morcos>
i just see both choices as non-optimal and i think we should pick one and try to make it as clear to users as possible
< gmaxwell>
This is a sign that our current definition is just broken. It should not be so tightly coupled to the mempool.
< morcos>
i thought that previously we had picked, and maybe failed at the making it clear
< jonasschnelli>
I'd say lets pick what serves more user... and the default = true seems to be the better choice...but I don't have numbers to proof that.
< bsm1175322>
One could create a RBF replacement transaction instead of a chain...
< gmaxwell>
(like how is the software even supposted to be usable to people that don't have a mempool? -- this is a supported configuration!)
< sipa>
gmaxwell: even if it is not tightly coupled with the mempool, we need a means of estimating whether it could confirm
< morcos>
yep, our time would be better spent extending bumpfee to work on chains
< gmaxwell>
or finding a way to eliminate the chain limit.
< jnewbery>
morcos: if the default is false (accept long chains) then it's very difficult to communicate to the user what the problem is. If we reject long chains then at least we can send a helpful error to the user
< sipa>
gmaxwell: i'm not convinced the chain limit itself is the only problem here
< morcos>
i suppose i do agree with gmaxwell thought that i always just took our balance calculation as gospel, but maybe it is kind of silly
< gmaxwell>
sipa: clearly not, because apparently we'll report a balance way off if you don't have a mempool! :)
< jtimon>
suggested topic: what's the current state on finally removing accounts?
< luke-jr>
bsm1175322: +1
< gmaxwell>
(er it's clearly not the only problem!)
< sipa>
gmaxwell: if you don't rely on the mempool, it's not that hard i think to make the wallet double count
< wumpus>
right, the wallet can't detect conflicting transactions itself
< sipa>
that would be great to fix, but i don't know how
< gmaxwell>
good thing there aren't any wallets in the bitcoin system without mempools.
< wumpus>
so if it sends a transaction, and someone malleates it and it would receive the malleated version back, it'd count that double
< morcos>
sipa: but a better balance calculation would be to evaluate net changes on a per tx basis
< sipa>
gmaxwell: wallets that don't spend unconfirmed change don't have this problem
< morcos>
and not consider the debits and credits separately
< wumpus>
it would work if it wouldn't count unconfirmed transactions
< sipa>
morcos: oh, you mean like the account system? *ducks*
< wumpus>
exactly, we have an option for that already
< gmaxwell>
sipa: no such wallet exists in the wild. (beyond bitcoin core users who have changed their settings and a few industrial users)
< morcos>
sipa: sigh.. no i mean properly.. there is no reason to assume a sent tx not in your mempool should debit your input but not credit your change output. thats just broken.
< morcos>
at worst it should do both
< morcos>
only gets complicated if its a mixed debit tx
< gmaxwell>
well it's showing a worse case balance, which is a thing you can rationally choose to do... but it's confusing to users esp with no other information available elsewhere.
< sipa>
gmaxwell: i don't know how to give an accurate unconfirmed balance without a mempool
< morcos>
actually, maybe gmaxwell is right.. maybe we can just fix that in our existing system?
< morcos>
gmaxwell: how is that worst case, how is that balance even achievable?
< gmaxwell>
Unfortunately, malleablity is still a thing.
< gmaxwell>
morcos: It's not achievable. But the estimation pattern of including non-mempool debits but not credits is a worst case estimator generally.
< morcos>
yes, but you can't end up with the debit and not the credit.. you can end up with the debit and you're momentarily confused how to spend the credit, but it's still your credit
< sipa>
without malleability maybe this problem becomes easier
< gmaxwell>
sometimes you have a non-mempool debit which will still go through.
< luke-jr>
sipa: are you including the wallet's storage of txs as "mempool"?
< sipa>
luke-jr: no
< gmaxwell>
I think any change estimation issue goes away if you assume non-malleablity and no concurrent use of the same keys.
< gmaxwell>
er balance estimation.
< sipa>
luke-jr: i mean a mempool which is kept consistent with the block chain - i guess you can simulate that inside the wallet, but it risks missing things that depend on unconfirmed transactions which don't involve you
< gmaxwell>
I find it hard to believe that the current behavior won't cause wildly wrong balances in other cases. In particular, what happens to your balance when you pay something that falls out of the mempool due to low fees? same deal.
< gmaxwell>
Chaning the behavior for long chains will do nothing for that, just covers up the fundimentally bad behavior.
< sipa>
right, the expected behaviour there is that you use abandontx to correct the balance
< gmaxwell>
maybe it's not reasonably possible to fix completely in the presence of malleablity. The best thing with malleablity still around might be presenting a pending balance.
< jtimon>
sipa: but there will still be malleability for old txs, no? I don't undesrtand the discussion well enough...
< sipa>
gmaxwell: maybe people just don't hit the "falls out of mempool" case, and only hit chain length limits
< gmaxwell>
sipa: well you can use abandon in this case too. (though thats a pretty bad expirence, spend a cent, then hours later 100 btc vanishes from your balance? )
< jonasschnelli>
jtimon: The main user issue is described here: #9752
< sipa>
maybe we should have some form of automatic abandoning...
< morcos>
sipa: noooooooooooo
< gmaxwell>
sipa: we previously had reports about txn falling out of the mempool.
< sipa>
or at least automatically stop counting as debit at some point
< gmaxwell>
sipa: AutoFraud(tm)
< jonasschnelli>
I agree with sipa, especially the non sendto* (or Qt) ones.
< wumpus>
fee bump is a better alternative to abandoning
< gmaxwell>
wumpus++
< jonasschnelli>
wumpus: yes. but adding new outputs would be a requirement then.
< sipa>
but if you stop counting as debit, while still excluding from unspent outputs, you risk even worse unexpected behaviour
< BlueMatt>
yes, better to not auto-abandon and do what other wallets are doing now - if you try to send with too low a fee, nag the user really loudly to make it rbf-able
< morcos>
i'd be happy to discuss another time whether we can make some slight improvements to our balance estimation.. i guess i think it wouldn't be that hard... next time i have time i'll look closer
< gmaxwell>
BlueMatt: yes, what electrum does is reasonable there.
< wumpus>
abandoning is dangerous, there is no guarantee that everyone forgot the transaction, so the user may send the tx again with different outputs and then it goes through twice oops
< jonasschnelli>
The reminds me of the problem that BIP125 doesn't explicit mention a recommended nSequence nr. Electrum was using 0, Core intmax-2. (privacy)
< sipa>
right, i take back my suggestion to auto-abandom
< BlueMatt>
auto-bump, otoh.....
< sipa>
yeah...
< wumpus>
sipa: from a viewpoint of the user it's what they want, for the transaction to 'just disappear', bitcoin just makes that very difficult
< morcos>
yeah auto bump should be 0.15 priority
< gmaxwell>
precomputed bumps with locktimes were always an idea I liked... doesn't really do great with spending unconfirmed change.
< jonasschnelli>
morcos: with plenty of pre-signed transactions?
< morcos>
spending unconfirmed change is doable i think... complicated, but you just stop bumping the first and start bumping the 2nd with CPFP calculations
< BlueMatt>
esp given miners can freely malleate it out from under you
< gmaxwell>
without malleablity basically none of these change handling issues would exist, I think.
< gmaxwell>
as you'd never have a case where you might double count your own funds.
< wumpus>
unfortunately we're stuck with malleability
< morcos>
not if we use flextrans
< morcos>
(sorry)
< sipa>
right, no from-self transaction in your wallet could credit you without you having signed for it
< gmaxwell>
hah
< jonasschnelli>
heh
< wumpus>
flextrans, lol
< BlueMatt>
trolol
< gmaxwell>
morcos: I made the remove all sending ability quip above!
< gmaxwell>
well we really haven't pushed to get malleablity fixed as a group... just put the fix out there.
< * sipa>
casually mentions segwit
< morcos>
ok... we're going off the rails. now.. maybe next topic.. and we revisit this in a week after thinking through both avenues
< sipa>
ok
< BlueMatt>
ack
< gmaxwell>
Sounds great.
< jnewbery>
yep
< jonasschnelli>
While we are touching the wallet... can we make progress on #9294?
< wumpus>
#topic status of removal of account system
< jonasschnelli>
I'd like to continue with HD restore... but 9294 seems to be required first
< wumpus>
but that it needs to happen is clear
< jtimon>
wumpus: oh, right, we need 7729 first
< sipa>
gmaxwell: no, i mean fix the silly continuous allocation of leveldb memory
< gmaxwell>
stevenroose: that is likely doofsus still signling 'bip 109' on testnet. (even though nothing implements it anymore)
< wumpus>
jonasschnelli: yes the HD chain split *defnitely* needs to be in 0.15
< gmaxwell>
sipa: so memory reduction not mempool reduction.
< wumpus>
jonasschnelli: it's sad it missed 0.14
< jonasschnelli>
wumpus: merging sooner should allow more perofmance improvemnts before 0.15.
< sipa>
gmaxwell: lol. yes
< jonasschnelli>
*performance improvements
< gmaxwell>
jonasschnelli: you can still implement lookahead scanning without the split.
< wumpus>
jonasschnelli: agree, will take a look at it
< stevenroose>
gmaxwell, yeah I read about bip109 as well when I googled the versionbit. So that means that 95% of testnett blocks the last few weeks were mined by people trolling about bip109?
< jonasschnelli>
gmaxwell: Yes. But I don't want to go to the rebase-hell. :)
< wumpus>
jonasschnelli: at some point we should merge something so that it can be improved
< wumpus>
right, it's frustrating to keep non-trivial things up to date with all the code churn
< luke-jr>
wumpus: I don't necessarily disagree with your points, just that they're factoring unrelated to multiwallet itself IMO
< achow101>
stevenroose: not now or here. ask in #bitcoin-dev, there's a meeting going on here right now
< jtimon>
I wouldn't mind some re-reviews on #8855 (previously #6907), it's simple
< gmaxwell>
stevenroose: no, it got 'activated' eons ago. then the miner signaling it mined BIP109 invalid blocks (because their implementation was broken) and forked classic off (until classic ripped out 109)
< jonasschnelli>
The whole Qt part is unsolved IMO.
< jonasschnelli>
The general concept of switching/opening/closing/creating wallets
< stevenroose>
achow101, my apologies
< luke-jr>
jonasschnelli: no, I have a branch for that
< wumpus>
jonasschnelli: I'd be happy to have it in JSONRPC already
< luke-jr>
jonasschnelli: it's 2 PRs away
< wumpus>
jonasschnelli: no need to block anything on GUI support
< jonasschnelli>
luke-jr: great!
< jonasschnelli>
wumpus: Sure...
< jonasschnelli>
But in general the low level stuff should conceptually fits the UI goals
< wumpus>
even small steps forward are worthwhile in this regard
< wumpus>
jonasschnelli: sure
< jonasschnelli>
Yes. Multiwallet was hold back long enought... I'm happy with every simple babystep
< luke-jr>
jonasschnelli: Knots has actually included multiwallet since 0.13, FWIW
< jonasschnelli>
luke-jr: Never used Knots. I probably should try it at least.
< wumpus>
ok, other topics?
< jnewbery>
running python rpc tests from `make check`?
< kanzure>
was someone asking about nulldummy versionbit?
< luke-jr>
isn't nulldummy deployed with segwit?
< sipa>
yes
< achow101>
I've been getting some reports about people's nodes running out of memory. perhaps we need to publish a "minimum spec" so people know what to expect if they don't meet that
< cfields>
jnewbery: +1. we were discussing this a few min ago. That makes "make check" dependent on python3 though (apparently). Not sure if wumpus is ok with that
< wumpus>
cfields: I don't mind
< wumpus>
cfields: the only thing I worry about is the slowness of the RPC tests
< jonasschnelli>
What's the benefits of adding the rpc's to `make check`?
< wumpus>
'make check' should ideally do fairly quick checks, some of the RPC tests classify as that, but the whole suite takes maybe too long
< gmaxwell>
make check is currently not quick at all.
< gmaxwell>
I think on my system it actually takes similar time to the whole rpc checks.
< wumpus>
gmaxwell: secp256k1 is part of that :p
< gmaxwell>
let me revise.
< cfields>
wumpus: same. But lately I've been coming around to gmaxwell's point that they're a bulk of our tests, so it's kinda a disservice for people to assume that "make check" and all is good
< gmaxwell>
the unit tests themselves take almost as long as the rpc tests.
< gmaxwell>
and are _far_ less useful.
< jonasschnelli>
Indeed. Adding another 20min rpc test will result in nobody running make check anymore
< wumpus>
it does the extensive tests for secp256k1, which take quite a while
< gmaxwell>
jonasschnelli: Does anyone but us run make check now? :P
< wumpus>
gmaxwell: that's certainly not true here
< cfields>
wumpus: also, this would parallelize the tests. So the boost tests and rpc would run at the same time
< jtimon>
jnewbery: I would prefer a diferent target, you could still do make check tests, or only make check or only make tests
< luke-jr>
I'd rather `make check` be comprehensive than quick tbh. the default RPC test suite seems like an okay compromise.
< wumpus>
yes, I run make check a lot
< jonasschnelli>
gmaxwell: I hope so... but i doubt.
< wumpus>
cfields: ok that's pretty cool
< gmaxwell>
wumpus: the secp256k1 tests are adjustable and can basically take as little or as much time as you like, we could make it arbitarily fast.
< jnewbery>
I could select a subset of fast rpc tests if you think the standard list is too slow
< jonasschnelli>
I guess not even the gitian system runs make check
< gmaxwell>
wumpus: though I'd like to move some more of the secp256k1 tests to runtime, it isn't like distributors actually make check. :(
< wumpus>
gmaxwell: I dont think running the secp256k1 tests thoroughly is a bad idea at all
< wumpus>
gmaxwell: helps catch compiler bugs and such
< jonasschnelli>
though a bit more complex because of the platforms.
< jtimon>
wumpus: maybe the tests to run with make should be all but excluding prunning.py?
< gmaxwell>
yes, I think they're important, though we could move some of that to simple startup time. The most critical checks are very fast.
< wumpus>
e.g. broken signing is very, very bad
< gmaxwell>
I worry a lot about compiler bugs, our current make check is woefully inadequate (except the libsecp256k1 part, granted. :P )
< jtimon>
cfields: but the unittests themelseves don't run in parallel like the rpc/py tests, right?
< gmaxwell>
wumpus: (similar to how I nagged you to make those rng tests runtime and you did...)
< jonasschnelli>
jtimon: not that i know
< gmaxwell>
(thank you)
< wumpus>
jtimon: parallelism at multiple levels doesn't make much sense, there's only so many cores to go around
< jtimon>
also, make check tests -j10 should pass -j10 down to the rpc-tester, right?
< cfields>
jtimon: correct, we'd never survive that
< gmaxwell>
wumpus: we could probably define a subset of rpc tests that are fast and more useful than the unit tests.
< cfields>
jtimon: ooh yes, that'd be really nice
< wumpus>
gmaxwell: yup. don't know if you saw the clang fsafe-stack issue that messes up deterministic signing
< jtimon>
wumpus: well, current make check could be faster, I compile very fast, but then it gests stuck at 1 core running the tests
< jtimon>
half the time I wait more for the unittests than to compile
< gmaxwell>
wumpus: I didn't.
< wumpus>
jtimon: but cfields proposes running (some of) the qa tests at the same time
< wumpus>
gmaxwell: let me dig it up
< gmaxwell>
Has anyone recently 'profiled' the tests to see where time is being spent?
< gmaxwell>
I bet we have cases where 20% of the time is checking if addition works or something. :P
< jnewbery>
I've profiled rpc tests. A lot of time is spent in stopnode()
< gmaxwell>
wumpus: holy fuck!
< wumpus>
both frameworks measure the time spent in every test, so profiling at that level is easy
< jtimon>
wumpus: well, I suggest a different target, but if they don't depend on each other, I guess they would run "at the same time"
< gmaxwell>
wumpus: good for tests. (but as I said, we should make some of those runtime too)
< wumpus>
I don't remember by heart which ones, though
< morcos>
i believe the rpc tests could also be made faster if tx relay had a different poisson distribution for regtest or something... i seem to remember that being an issue
< cfields>
wumpus: whoa. Isn't that default for clang now? Or proposed, at least?
< gmaxwell>
regardless of the specific example, compiler bugs are a real thing.
< wumpus>
cfields: I think it's going to be more widely enabled, yes, though AFAIK not yet. I only caught it because cloudabi already has it as default
< cfields>
jnewbery: i'd be interested in your findings there
< gmaxwell>
(though seeing them in rather boring C code is depressing)
< gmaxwell>
wumpus: in any case thats the best news all day! I've complained many times that our tests must suck because we've not found any miscompliation bugs.
< wumpus>
(test_bitcoin -log_level=test_suite shows which unit tsts take so long. most are really fast! )
< gmaxwell>
wumpus: finally some evidence that our tests are potentially okay. :P
< wumpus>
gmaxwell: heh
< jnewbery>
ok, sounds like there's no fundamental objection to at least doing some rpc tests in make check. I'll open a PR and we can continue discussion there.
< wumpus>
gmaxwell: and yes doing some quick secp256k1 tests at runtime would make a lot of sense
< wumpus>
gmaxwell: basic sanity is fairly easy to check
< gmaxwell>
jnewbery: yes, and we should look at time measurements and rebalance the tests to be more useful.
< jnewbery>
also, once 9956 is merged we can stop calling them rpc tests!
< gmaxwell>
wumpus: I have a branch somewhere that adds some runtime selftests, but I think I got a bit carried away and put it aside. :P
< jonasschnelli>
jnewbery: Yes. I'd like to see that merged.
< wumpus>
jnewbery: no, no fundamental objection. Just about speed but that doesn't depend on the language/framework
< wumpus>
jnewbery: +1
< jtimon>
jnewbery: I still heard no reason against adding a new target instead of reusing check
< wumpus>
some of the qa tests are really fast, some of the unit tests really slow, indeed should rebalance testing bang for buck
< jonasschnelli>
wumpus: If you run with wallet, you have a basic EC sanity check over VerifyPubKey().
< jonasschnelli>
But we should probably extend it and add call it from non-wallet init parts
< jnewbery>
jtimon: I like the idea of `make tests` for the full suite, but I think `make check` should include some rpc tests
< jtimon>
jnewbery: oh, I had missed 9956, thanks for pointing it out
< gmaxwell>
jonasschnelli: a special test function could get a much more comprehensive test in not much time.
< cfields>
jnewbery: do you happen to still have any of those profiles showing slow stopnode() ?
< jtimon>
jnewbery: mhmm, why not make tests includes some python tests (maybe most of them) but make check none of them?
< jonasschnelli>
gmaxwell: I really wonder now why we haven't added a simple runtime EC sanity test...
< jnewbery>
jtimon: I've got some nits from cfields to address on 9956. Very happy for other people to review after that
< jnewbery>
cfields: I believe I have them lying around somewhere. Let me dig them out
< jonasschnelli>
I mean coffee machines do "runtime tests" and a faulty EC subsystem can cause far more troubles
< cfields>
jnewbery: thanks. I would assume that it's just waiting for the threads to stop. But I'm curious to see what they're spinning on
< jtimon>
so no answer? everybody seems to prefer reusing make check with some tests, but nobody seem to be able to explain why...
< jnewbery>
cfields: I think it just takes a long time (a few seconds) to stop a node. They're synchronous calls and stopnodes() will stop nodes in series rather than parallel. Some tests also do multiple stop-starts. It adds up.
< jonasschnelli>
The crazy think with the SafeStack issue in LLVM4.0 is that they are not willing to fix it for 4.0. It will be fixed for 4.0.1.
< jnewbery>
jtimon: patience :) I have no particular objection to either. What do you think the different use-cases are for `make check` and `make tests`? (ie under what circumstances should users *not* want to run a few quick python tests)
< cfields>
jnewbery: oh wait, you mean profiling on the python side?
< jtimon>
jnewbery: mostly giving the user more control on how much time he wants to spend running tests
< jnewbery>
yes. Oh, sorry you want profiling of the node's doing as it stops? I don't have that.
< cfields>
jnewbery: sorry, CConnman::Stop() used to be StopNode(). I thought that's what you were referencing.
< jtimon>
maybe it is the unittests they don't want to run for whatever reason
< instagibbs>
jtimon, some people don't know the rpc tests exist, or think that rpc tests run when you call it
< jnewbery>
`make check` currently runs the unit tests. You'd change that so it doesn't run unit tests either? What would there be left for it to do?
< jnewbery>
cfields: sorry name collision
< jtimon>
instagibbs: and those people run make check?
< instagibbs>
jtimon, this happened just last week, so yes
< instagibbs>
from someone I expected knew this
< jtimon>
jnewbery: no, I mean those people could run "make tests" and run only the python tests, or "make check" only the unitttests or "make check tests" to run both
< instagibbs>
(n = 1 and all that)
< jtimon>
instagibbs: I think this can be fixed with docuentation about the new test target and the current check one instead of change check to match their expectations (which I assume was that all python tests were run with check, something nobody seems to be proposing)
< jtimon>
I just don't see the advantage in giving the user less control, sorry
< jnewbery>
jtimon: I think the shorter form `make check` (ie the one that most people will type) should run a cross section of all types of test (ie all unit plus some python). I'm not opposed to having other forms that run just the unit tests or just the python tests.
< jtimon>
make check is not shorter than make test, but whatever, why not think of another name for the new functionality and leave the option that only run unittests (if we're going to have one) with its current name?
< instagibbs>
jtimon, I don't really care a lot, just saying this problem does actually seem to exist
< jtimon>
anyway, I guess this is not so important to disccuss it so much
< jnewbery>
I also don't care too much on what we call it :)
< jnewbery>
but I'll make sure there's a way to just run unit tests
< jtimon>
instagibbs: just stating my preferred solution
< jtimon>
jnewbery: cool
< bitcoin-git>
[bitcoin] jnewbery opened pull request #10017: [POC] combine_logs.py - aggregates log files from multiple bitcoinds during functional tests. (master...log_aggregator) https://github.com/bitcoin/bitcoin/pull/10017