< paveljanik>
maybe. But the similar problem was reported by kwin people. So I think it is generic.
< paveljanik>
will download other OS binary to see.
< jonasschnelli>
missing .pc file will probably break all platforms?!
< wumpus>
well the .pc files were broken on MacOSX in some cases, maybe they've removed them because of that
< paveljanik>
maybe they are missing on OS X only. Do not know. I'm checking Linux now.
< wumpus>
I would be really surpised (and disappointed) if they removed them on unix/linux
< wumpus>
there's not really a replacement for pkgconfig on linux
< paveljanik>
they probably expect projects to use cmake and their files.
< wumpus>
(well ok, manually specifying all the directories)
< wumpus>
cmake uses pkgconfig too IIRCI
< wumpus>
cmake is just an autoconf replacement, it doesn't replace pkg-config
< paveljanik>
OMG 661MB...
< wumpus>
pkg-config is not part of any specific make system, it is just linux's way of locating development headers and libraries, OS X has this 'framework' system instead
< qwebirc646325>
does anyone know if its possible to fork bitcoin to createe a new coin with support for sending 1 satoshi
< qwebirc646325>
or if a fork could elimnate transaction fees
< wumpus>
qwebirc646325: altcoins can make whatever change they want
< wumpus>
although both examples you meantion aren't even consensus rules. Any miner could accept transactions of one satoshi, or accept transactions without fee (some even do).
< qwebirc646325>
great
< qwebirc646325>
how much needs to be changed in the source to create an altcoin
< wumpus>
this is not the place for altcoin questions
< btcdrak>
##altcoin-dev
< JackH>
come on guys, the altcoin might become the biggest thing ever!
< JackH>
you dont know what you are missing out on
< wumpus>
hah
< JackH>
nice on BIP9 btw :)
< morcos>
gmaxwell: wumpus: sipa: sorry for being a pest, but I would like some direction on this getbalance mess. does one of you want to talk it through with me?
< wumpus>
morcos: at least ignore the accounts stuff
< wumpus>
yes, balances can be off with accounts, we know that
< sipa>
morcos: i've been thinking about it
< sipa>
and i wonder how it was really ever intended to work
< morcos>
wumpus: well. that would lead to us not actually solving the problem that Cocodude first brought to us
< sipa>
wumpus: it's not that simple
< morcos>
also i'm most concerned by the fact that the account balances are what is used for sendfrom and sendmany
< sipa>
morcos: ??
< sipa>
wumpus: the account balance calculation is very strongly related to the computation of transaction 'effects' (which is what listtransactions lists)
< morcos>
those functions get the available balance by calling GetAccountBalance
< sipa>
what?
< wumpus>
I don't think so, the send functions allow balances to go negative
< wumpus>
(account balances)
< sipa>
morcos: ... you're right, i thought that was changed in... 0.3.x
< morcos>
my proposal was to just special case check if the given account is "" and then not use GetAccountBalance
< wumpus>
people are finally looking at the wallet code in detail, that's good to hear :)
< morcos>
but this is getting to be an invasive change for a point release
< wumpus>
yes, try to solve it for master at least
< sipa>
any correct solution is going to result in the account balances being correct anyway
< morcos>
sipa: or removing accounts?
< sipa>
if you can't get account balances right, listtransactions will also be wrong
< sipa>
because they use the same code
< wumpus>
maybe it's just too much of a change for a point release, that's a fair conclusion
< morcos>
what do you mean by listtransactions being wrong?
< morcos>
wumpus: well the problem is we probably ought to do something. there is a problem in the released code now. its a matter of deciding what we should do for 0.12.1. then there is a second question of what we should do for master.
< morcos>
are we at the point where we can remove accounts? if so, that is what we should do for master, b/c anything else is just a recipe for further problems down the road
< wumpus>
yes, we should definitely remove account balances
< morcos>
to be honest, i'm not really interested in taking the time to understand the accounting system well enough to try to make it work properly. (since i don't believe its something we want to long term support)
< wumpus>
people use other account functionality (e.g. they use them as labels)
< wumpus>
but the balance logic is incorrect and dangerous
< morcos>
wumpus: so my suggestion is if you don't really intend to be using account balances but you just are b/c thats the only way to get the total sum balance, then we should instead make you not use account balances
< sipa>
morcos: there are two ways to look at the wallet; 1) as a set of utxos 2) as a sequence of balance updates
< morcos>
this exists in at least 3 places now
< morcos>
1) getbalance (where you want to specify a confirm requirement you have to fill in an account argument), sendfrom, sendmany
< morcos>
sorry, that was all 3
< sipa>
morcos: effectively, you either iterate over the transactions to find which of its outputs are available
< sipa>
morcos: or you iterate over transactions to see which utxos they add/remove
< sipa>
the first is used by listunspent and getbalance "*"
< sipa>
the second is used by listtransactions and getbalance acc
< morcos>
sipa: yes, but i think thinking of it as a sequence of balance updates is fairly complicated when sometimes you want things to count for reducing your balance but not for adding to it. (in the case of unconfirmed txs)
< morcos>
sipa: actually getbalance "*" is more similar (but a separate code path) to getbalance acc i think
< sipa>
morcos: my point is that if you can't compute balance updates correctly, listtransactions will be wrong
< sipa>
because listtransactions does not list transactions, but balance updates caused by transactioms
< sturles>
At least some people use accounts. If you remove accounts in the segwit release, it may impact the upgrade speed negatively. E.g. I will have to code an entirely new solution for my system before I can use a version without accounts.
< morcos>
sipa: i'm not sure i understand still. i think it'll list all the balance updates as you say. what it won't do is provide you an intelligent way to add them up to arrive at something meaningful
< morcos>
but i think each individual one would make some kind of sense depending on how you look at it
< morcos>
the question comes when you are trying to decide whether you want to count the balance update in your total or not, but with listtransactions, you don't have to make that choice
< wumpus>
sturles: I'm talking about removing account balances, not remove all the account related RPCs
< sipa>
sturles: those are independent; if we make any meaningful changes to the account system, it will be in a major release and not backported
< morcos>
sturles: yes we wouldn't remove accounts for segwit release, that would be for a major version
< sipa>
sturles: consensus changes like segwit are always backported
< wumpus>
account balances are unreliable, the other parts are not.
< morcos>
sipa: did that make sense what i just said? i'm not sure what you think will be "wrong" about listtransactions
< sipa>
morcos: both use CWallet::GetAmounts
< morcos>
yes, but i think the problem is in the filtering of whats returned from GetAmounts and listtransactions doens't filter it
< sipa>
hmm
< sipa>
ok
< sipa>
but we have so many different states a transaction can be in
< sturles>
Much of my system relies on account balances beeing correct. :-/
< sturles>
Account balances have always been reliable for me.
< wumpus>
that's very dangerous
< sipa>
wumpus: i think account balances are perfectly well defined
< morcos>
sturles: i think it had some broken behavior in 0.11 and it has some other broken behavior in 0.12
< wumpus>
sipa: I've heard otherwise
< sturles>
Oh?
< wumpus>
sipa: I certainly wouldn't recommend anyone to rely on them
< sipa>
they have unexpected behaviour wrt unconfirmed transactions
< sturles>
Is there a bug report I can read?
< wumpus>
the thing is, no one really understands it, it's just too messy
< midnightmagic>
Before I retired it, I had a wallet where almost all the accounts had negative balances.
< morcos>
wumpus: +1
< wumpus>
did you read the convo between morcos and sipa above?
< wumpus>
those are two long-term developers, who have worked on the code for a long time, and they're surprised about how some of the account things work - doesn't that say enough?
< sipa>
i think we should go over the possible states a transaction can be in, and think about what their effect on listtransactions, listunspent, and getbalance should be (independent from accounts)
< wumpus>
the problem is also that whatever broken behavior accounts have, people may be relying on it, even if it's unknown to us
< sipa>
1) confirmed 2) unconfirmed but in mempool 3) unconfirmed not in mempool 4) unconfirmed not in mempool, abandoned 5) unconfirmed not in mempool, known to conflict
< morcos>
sipa: ok. are you narrowing the discussion to differences in behavior between 0.11 and 0.12?
< sturles>
I can't say I understand how the code works, but I do understand how accoutns work. Negative balances are not a problem. I use negative balances as well, in my accounting.
< sipa>
morcos: no, i first want to know what we think the ideal behaviour should be
< morcos>
sipa: and when you say getbalance, you have to refer to what arguments you are using. btw, there is also getunconfirmedbalance
< wumpus>
and I'm sure there is previous discussion
< sipa>
morcos: ok, so we should treat those separately
< sipa>
also, there is immature
< morcos>
sipa: well i'm just trying to avoid the rabbit hole of trying to fix accounts perfectly. i htink a better goal is to avoid regressions, and work towards removing accounts
< sipa>
and non-final
< wumpus>
morcos: +1
< sipa>
morcos: i'm not trying to fix accounts
< midnightmagic>
sturles: The wallet in my case had a total balance adding the accounts up, to a value different than the listunspent returned.
< sturles>
wumpus: Yes, malleability issues messed with my accounting as well back in 2014, but I didn't have the problem during the last malleability attack where someone changed lots of transactions.
< sipa>
morcos: i want to know what we think the correct behaviour should be for those different types of transactions, on listunspent, getbalance, getunconfirmedbalance, listtransactions
< sturles>
midnightmagic: OK, I can't explain that.
< sipa>
no account-related calls in that list
< wumpus>
midnightmagic: yes that was quite common at a certain point
< morcos>
ok, so if you don't want to worry about calling getbalance("", n) or getbalance("*", n) or what gets used in sendfrom, sendmany, then i don't think its that hard
< sturles>
midnightmagic: Some transaction cleaned out (abandoned) perhaps?
< wumpus>
the account system has pretty much been unmaintained from 2011-2012 or so
< morcos>
the thing we discovered yesterday, was that tx type 5 in your list above is not necessarily distinguishable from tx type 3
< midnightmagic>
gmax tried to help me debug it, but at some point I abandoned the code and that wallet and rebuilt everything and respent it. Still not more than 95% sure I swept it all up.
< sipa>
morcos: yes, i know
< morcos>
this is a crap ton of stuff to write up. i think i can do that reasonably well for 0.11, 0.12 and proposed fix
< sipa>
morcos: so presumably we want to treat 3 and 5 the same, apart from reporting
< morcos>
great. agreed with that
< wumpus>
yes
< sipa>
... which means that the introduction of 5 was maybe overkill
< sipa>
though i guess it can be useful for example for the gui to be able to suggest abandoning
< morcos>
oh wait, sorry
< morcos>
i don't actually agree we treat them the same
< wumpus>
sipa: in some cases you may also want to treat transactions that you sent yourself differently from received ones
< morcos>
exactly
< morcos>
if you are considering whether your inputs are spent. 3) considers them spent, 5) doesn't
< wumpus>
(I miss that in the list, but maybe that's a cartesian product)
< morcos>
sorry whether your available coins are spent
< morcos>
if you are considering whether you have new available coins to spend 3) and 5) should both be no you don't
< wumpus>
e.g. the wallet could create a transaction, but you have wallet broadcasting disabled
< wumpus>
you'd still want it to subtract from your balance and hold the inputs, at least until abandoned
< morcos>
yes
< morcos>
let me write all that up in detail, i can do that. what i want to know is a proposed solution for getbalance("", n), sendfrom, sendmany ? in particular the problem reported to us was a result of getbalance("",0)
< wumpus>
in any case, making all of this consistent is too much for a point release, this would be something for a major release (+mention in release notes)
< sipa>
sturles, midnightmagic: since you guys use/used the account system, were you relying on sendfrom/sendmany failing when you send from an account with a too low balance?
< morcos>
wumpus: actually the code changes are going to be small. probably just the first commit on 7706, plus potentially this question of skipping account accounting and using global balances when the account is "" in those 3 rpc calls
< sturles>
sipa: I did at some point, but not now.
< sipa>
that's the one thing that surprises me today, finding out that they do a balance check
< sipa>
because there are several other ways in which account balances can ge negative without any protection
< sturles>
Yep, especially the "" account.
< morcos>
sipa: oh, that hadn't occurred to me, that it wasn't important
< sturles>
Otherwise it is mostly due to fees.
< sipa>
morcos: well, maybe it is, and i just don't know!
< morcos>
well that makes things a lot easier, then i would suggest we don't change anything other than the first commit in 7706 and we can tell people that if they want total unconfirmed balance they should call getbalance() + getunconfirmedbalance() and not use getbalance("", 0)
< morcos>
that would make me happy. minimal changes.
< sipa>
morcos: it was an intentional change at some point very long ago (i believe in the 0.3.2x days), to not protect against account balances going negative, because it wasn't even possible
< morcos>
its only through my stupidity in not knowing how getbalance("", 0) works that we even realized there is a problem in the wallet.cpp getunconfirmed balance code.
< midnightmagic>
sipa: I've been using only the rawtx interface for too long to remember my use of accounts. gmax told me early on to stop using it. i was lazy and never cared if the send* failed or not for whatever reason. post-accounting aggregation has never been a concern for me. :-( Sorry man.
< sipa>
morcos: for example, the move RPC has a 'minconf' argument that is ignored
< morcos>
yeah i didn't even realize that RPC existed until yesterday
< sipa>
morcos: it used to check whether there was enough balance in the account being moved from, at the given confirmation level
< midnightmagic>
sturles: my accounts went into the thousands negative
< sipa>
morcos: so if there is any change we make to this, i'd say we remove that check entirely...
< morcos>
so i do think there was a regression in 0.12 in how bad account balances are as a result of this business with unconfirmed txs... but maybe we should just let that be, other than communicating it
< morcos>
sipa: my concern was when you weren't trying to use it for accounts! isn't the only way to sendmany to do sendmany with the "" account
< sipa>
right
< sturles>
You cam make the numbers negative with move as well, of course. In the millions negative.
< sipa>
sturles: yeah, that's exactly what i was saying above, move and sendtoaddress don't check balance
< sipa>
so why would sendfrom/sendmany?
< sturles>
I have used that trick a few times, to make enough funds available in the account I was sending from. :-)
< morcos>
sipa: ah ok. now i see. i was worried that sendmany with a "" account would create transaction that spent too many funds. but it won't. CreateTransaction will stop it.
< sturles>
I see no reason why sendmany or sendfrom should check the balance of an account, but make sure to make a strong note of it in the release notes..
< morcos>
thats why i was concerned about it.
< sturles>
Alternatively make it an option to check the balance.
< sipa>
sturles: which is called getbalance :)
< morcos>
I think we should just make minimal changes and announce a removal timeline for accounts. Seems a lot for 0.13, maybe 0.14
< morcos>
I'll put it in a fresh PR, so its cleaner
< GitHub7>
[bitcoin] morcos opened pull request #7715: Fix calculation of balances and available coins. (master...fixconflicts_take2) https://github.com/bitcoin/bitcoin/pull/7715
< GitHub85>
[bitcoin] morcos closed pull request #7706: [WIP] Fix calculation of balances and available coins. (master...fixconflicts2) https://github.com/bitcoin/bitcoin/pull/7706
< morcos>
sipa: wumpus: ok checkout #7715, i _think_ that chart is right.
< wumpus>
nice work!
< morcos>
probably needs someone to make sure its right though
< btcdrak>
morcos: <3 that chart
< sipa>
morcos: awesome!
< sipa>
(i had no idea github had so many icons...)
< morcos>
I perhaps did not make it clear enough that the red triangle can't lead to bad things. In other words you won't reduce your balance for coins spent if those were coins that weren't included in your balance.
< morcos>
The fearful face can I think lead to bad things though.
< sipa>
morcos: use :arrow_down: instead of the red triangle?
< sipa>
and :warning: for the unhappy face?
< morcos>
ah, good arrow down is better, but i like the fearful face. you should be fearful!
< jtimon>
now that I have a good computer...can't the rpc tests be parallelized?
< jtimon>
at least for people with zmq installed or something
< jtimon>
just thinking out loud, don't take this too seriously yet
< sipa>
i don't see how zmq is relevant for that
< sipa>
you could run multiple tests in parallel though, just running different bitcoind's in different directories side by side
< jtimon>
yeah, forget about that, just that I like to use zmq for concurrency in python, sorry for bringing that up
< jtimon>
to your second comment, yes, that's what I was thinking, but with -j56 instead of having to think about it, it was just a wish in the open
< jtimon>
to be perfectly clear, the goal is running ```python ./qa/pull-tester/rpc-tests.py -extended -j4``` or something like that
< jtimon>
but as said is just a random thought
< sipa>
that sounds totally reasonable and doable
< jtimon>
zmq is just the way I would support that, so totally forget about if you don't like zmq/nanomsg
< jtimon>
for me, messaging is the simplest way to levereage both threads and processes transparently
< sipa>
there is nothing to communicate even
< sipa>
just run multiple tests at the same time, and make sure they use separate directories/ports
< jtimon>
yeah, if anybody has a script that does that, please share. If I ever write it myself (which would still be prefarable to me than your "this can be done relatively easily manually"), I would do it using python and zmq, but as said that's just an irrelevant side-note
< jtimon>
in the meantime python ./qa/pull-tester/rpc-tests.py -extended is not all that bad
< jtimon>
anyway, just wishful comments while I learn more about our wonderful testing setup
< jtimon>
rpc python testing newbie here
< jtimon>
still
< jtimon>
but that is good, that means unittests alone captured most of my stupid thoughts previously
< jtimon>
and I have a good computer to start testing other people's things more deeply, sorry for the distraction, shouldn't think out loud here
< sipa>
jtimon: i really don't understand where zmq comes in
< sipa>
i'd be in favor of implememting multithreaded testing in rpc-tests.py
< sipa>
using a -j flag like you suggest
< jtimon>
just for concurrency, seriously, just forget about that whole part, I shouldn't have mentioned it, there's 3000 other ways to do concurrency in python
< sipa>
ok :)
< jtimon>
yeah, the -j option is the whole point
< sipa>
but zmq is for communicating between processes, what do you expect to communicate?
< sipa>
anyway, nvm :)
< sipa>
if you feel like implementing it, and feel like zmq is useful for that, please do :)
< jtimon>
zmq hs many use cases than you think, I think, but I'm happy that we have decoupled the topics
< jtimon>
s/many/more
< jtimon>
but yeah, nvm
< jtimon>
I mean, if I implement it (which will depend on how often I run ./qa/pull-tester/rpc-tests.py -extended from now on) I will use that, and if anybody else does before me, the script can be written in php for all I care
< jtimon>
TLDR; catching up on testing, I can't remember last time I gave a full ACK instead of just an utACK for something that wasn't obviously correct to me, oh, wait...that should never have happened, I'm virgo that way :p
< btcdrak>
wumpus: 7716 should be tagged consensus
< btcdrak>
morcos: BIP is merged, you can send that email
< btcdrak>
wumpus: 7543 also needs to be tagged consensus now
< morcos>
btcdrak: email sent
< wumpus>
hm this is interesting, ubuntu 16.04 doesn't install python 2 by default anymore
< wumpus>
no /usr/bin/python nor /usr/bin/python2
< wumpus>
this breaks 'make check'
< wumpus>
it is possible to install python 2.7 using 'apt-get install python2.7` but this will only give you a /usr/bin/python2.7, no /usr/bin/python nor /usr/bin/python2...
< wumpus>
this is fucking annoying as it effectively makes it possible to refer to it as interpreter at the top of scripts
< wumpus>
impossible*
< btcdrak>
RIP python 2
< wumpus>
yea
< btcdrak>
wumpus: wait, do you have a time machine? It's only 16.03 last time I looked...
< wumpus>
yes, I inherited satoshi's delorean
< wumpus>
(you can find beta images for ubuntu 16.04)
< sipa>
damn
< sipa>
ubuntu 16
< sipa>
what happens to the past decade?
< sipa>
i think i started using ubuntu in 2006
< wumpus>
heh, me too, around 2005-2006, where goes the time
< wumpus>
before that I used gentoo and before that slackware
< sipa>
debian, gentoo, ubuntu here
< paveljanik>
slackware, red hat, suse, os x
< Luke-Jr>
kinda our fault for still using Python2..
< cfields>
wumpus: yea, that's really annoying. Is there no convention for a python2/python3 shebang, at least?