< bitcoin-git>
bitcoin/master 0e85204 Pieter Wuille: Add serialization for unique_ptr and shared_ptr
< bitcoin-git>
bitcoin/master da60506 Pieter Wuille: Add deserializing constructors to CTransaction and CMutableTransaction
< bitcoin-git>
bitcoin/master 1662b43 Pieter Wuille: Make CBlock::vtx a vector of shared_ptr<CTransaction>
< bitcoin-git>
[bitcoin] laanwj closed pull request #9125: Make CBlock a vector of shared_ptr of CTransactions (master...sharedblock) https://github.com/bitcoin/bitcoin/pull/9125
< gmaxwell>
\O/
< BlueMatt>
sipa: did you ever benchmark how much of #9125's performance improvement was actually just the #9014 stuff? (I assume a lot, though the real benifit is compact block speedups here, which reindex cannot test)
< bitcoin-git>
bitcoin/master 4662553 Cory Fields: net: don't send feefilter messages before the version handshake is complete
< bitcoin-git>
bitcoin/master c390640 Wladimir J. van der Laan: Merge #9117: net: don't send feefilter messages before the version handshake is complete...
< bitcoin-git>
[bitcoin] laanwj closed pull request #9117: net: don't send feefilter messages before the version handshake is complete (master...feefilter-assert) https://github.com/bitcoin/bitcoin/pull/9117
< jonasschnelli>
wumpus: do you have plans to rebase #7729? Otherwise I can pick it up.
< wumpus>
yes, I'll get back to that at some point
< wumpus>
happy some people are finally commenting on the API
< jonasschnelli>
Yes. Thanks to last weeks meeting.
< instagibbs>
wumpus, I'm not an old timer, which is why I'm asking more basic account questions. It's always been deprecated in my Bitcoin lifetime :P
< wumpus>
right, it's always been deprecated, it's time to move forward there
< wumpus>
"always"
< instagibbs>
But my assumption is correct, yes? That accounts gave balances which were often screwy/inconsistent, people should be doing this at higher layers by getting lists of outputs under labels or whatever?
< wumpus>
yes, it's a mess, people have shot themselves in the foot using it many times, and also most expect it to be something else than it is (eg "subwallets" with their own utxos)
< wumpus>
it's indeed not something that belongs at the wallet level
< instagibbs>
Yes I recall finding the comment in code about using * was incorrect in retrieving same balances, but I couldn't even describe the replacement text and gave up
< wumpus>
so should the argument to importaddress and sendtoaddress etc be called "address" or "bitcoinaddress"? we use a mix of both, which isn't very consistent
< wumpus>
same with importprivkey which uses "bitcoinprivkey" as argument name, but "importpubkey" which just uses "pubkey". Should make these consistent for #8811
< bitcoin-git>
[bitcoin] ryanofsky opened pull request #9196: Send tip change notification from invalidateblock (master...pr/notify-tip) https://github.com/bitcoin/bitcoin/pull/9196
< thermoman>
hi. I've got a transaction that's somehow not relayed to the network. but when doing getrawtransaction and sendrawtransaction afterwards it's relayed to the network
< thermoman>
is this a known issue somehow?
< sdaftuar>
thermoman: is this for a brand new transaction you created, or an old wallet transaction?
< thermoman>
ok, it just was relayed to the network - but a hughe delay
< thermoman>
sdaftuar: it's a brand new tx
< thermoman>
we have noticed big delays since 2 or 3 days
< sipa>
thermoman: define huge?
< thermoman>
doing sendrawtransaction as a workaround after the initial TX the TX was then immediately visible
< sdaftuar>
thermoman: can you clarify how you're telling that the tx is "visible"?
< sdaftuar>
ie, visible on some 3rd party explorer?
< sdaftuar>
or are you looking at your node's relay behavior more directly?
< thermoman>
lemme get that info from our dev
< thermoman>
I guess 3rd party
< thermoman>
like blockchain.info
< thermoman>
i query the dev right now
< sipa>
also, what kind of delays are we talking about? second? minutes? hours?
< thermoman>
the node in question is connected to a proxy node via connect=<IP>:8333
< thermoman>
it seems like the connection to the proxy node got reset
< thermoman>
all nodes run 0.13.1
< sdaftuar>
right, so you had no peer to broadcast to initially
< sdaftuar>
which would explain the delays you're seeing, though it raises the question of why you're getting disconnected
< thermoman>
right. the proxy node is on the LAN
< sdaftuar>
if you can share the debug log info around the time of a disconnect, that might help indicate if it's a bitcoind issue
< thermoman>
you mean from the proxy node, right?
< thermoman>
the proxy node isn't running with debug=1 - will have to change that
< sdaftuar>
well it might be helpful to see it from both sides' point of view
< sdaftuar>
if you just have one debug log, we can look at that
< thermoman>
lemme activate debug=1 on the proxy node. and when the error occurs again tomorrow i'll put both debug logs on pastebin
< thermoman>
ok?
< sipa>
thermoman: i believe you were not even connected to your proxy node when the transaction was creates
< sipa>
oh, sdaftuar already told you thay
< sdaftuar>
thermoman: does your proxy node -whitelist your internal node?
< thermoman>
sdaftuar: yes
< thermoman>
# Whitelist internal networks
< thermoman>
whitelist=a.b.c.d/24
< thermoman>
from bitcoin.conf on proxy node
< thermoman>
we now have debug=1 on proxy node. we'll monitor the debug.log from the proxy for new connections from our nodes and when this happens again I will come back here
< thermoman>
thanks so far
< sdaftuar>
ok, sounds good. if you have the debug.log from your internal node available where you saw the disconnect, we should be able to tell if your internal node decided to disconnect your proxy for some reason.
< sdaftuar>
also, if the proxy rejected something your node sent it, causing a disconnect, we would be able to see that as well, i thnk
< thermoman>
ok
< thermoman>
here is the interesting part from the node behind the proxy
< thermoman>
this is the stripped down log from the node sending the TX (behind the proxy)
< thermoman>
the issue occured several times after importing a privkey
< sdaftuar>
ah. i wonder if your node was too busy to respond to ping messages, causing a disconnect?
< thermoman>
that might be
< thermoman>
it seems the import did block everything else
< thermoman>
and directly after the import the TX was created (while being disconnected from the proxy)
< thermoman>
should timeouts be upped on our nodes?
< sdaftuar>
the second disconnect seems a bit different though
< thermoman>
yeah
< thermoman>
it's directly after
< thermoman>
Erased 100 orphan tx from peer 1
< thermoman>
should the nodes behind the proxy whitelist the proxy-ip, too?
< thermoman>
at the moment only the proxy has whitelisted the client ips
< sdaftuar>
thermoman: i don't think it would hurt, though i'm not sure yet what's going on
< sdaftuar>
ok i guess the interesting thing about the second disconnect is that your node tried to connect to it at 15:28:07, and then didn't send the version message untl 15:51
< sdaftuar>
so this looks like the wallet operation is contending with the p2p code, and network timeouts are being hit
< thermoman>
ah ok, didn't miss the connection between "Added connection" and "sending version message"
< thermoman>
s/didn't/did/
< thermoman>
current workaround for us is to wait a minute after importing is done
< sdaftuar>
thermoman: seems like a reasonable workaround for now. thanks for reporting this, i think this is something we should look into and improve
< thermoman>
do you add an issue in github or should I?
< thermoman>
we've only seen this issue after upgrading the proxy node von 0.11.2 to 0.13.1
< thermoman>
the node with the TX was running 0.11.2 until some hours ago and we noticed this behaviour with 0.11.2 and even after upgrading the node to 0.13.1
< thermoman>
so it seems to have started after upgrading the proxy node
< thermoman>
if I don't hear from you I'll open an issue on github tomorrow
< sdaftuar>
hmm, this issue really seems like it's specific to the internal node.
< sdaftuar>
please go ahead and open the issue, thanks!
< thermoman>
we never had this with 0.11.2 on both nodes
< thermoman>
(or didn't notice?)
< sdaftuar>
my memory is failing me right now about when and whether we've made changes from 0.11 to 0.13 with locking/thread contention. so in my view it's possible there's a performance regression, i'd need to investigate more.
< thermoman>
ok, thanks so far. will open the issue tomorrow
< sdaftuar>
thermoman: do you often call importprivkey in your setup, ie you were doing that on 0.11.2 as well as regularly on your 0.13.1 nodes?
< sdaftuar>
it looks to me like that is always a slow operation, and will cause bitcoind to be unresponsive while rescanning
< sdaftuar>
(and i think this would have been true in 0.11.2 as well)
< gmaxwell>
sdaftuar: importing a private key with rescan caused timeouts and disconnects for me w/ 0.12 too. If thermoman saw a change it could have also just been a bit of chain growth taking it above threshold.
< sipa>
gmaxwell: or rescans became a bit slower since
< sdaftuar>
instagibbs: in #9194, wouldn't it make sense (if we're going down this road) to also support legacy serializations for zmq and rest?
< gribble>
https://github.com/bitcoin/bitcoin/issues/9194 | Add flags to getrawtransaction and getblock to return non-segwit seri… by instagibbs · Pull Request #9194 · bitcoin/bitcoin · GitHub
< instagibbs>
sdaftuar, yes, I have literally no experience with those interfaces, what would that entail
< instagibbs>
fwiw I think the number of spots we'll need this for is basically two, the ones I've done. We support backwards compatbility when talking to non-segwit nodes(and maybe future serializations!), I'm just giving an option for that at rpc level.
< instagibbs>
so whatever makes sense for that
< sdaftuar>
instagibbs: see for instance rest.cpp:371
< sdaftuar>
oh wait, i made a mistake
< sdaftuar>
oh maybe not -- sorry i've never really looked at this code. looks like that function gets a transaction and serializes it for a user
< sdaftuar>
similarly in the rest_block() function in that same file
< sdaftuar>
and then i think there are two analogous places in zmq/zmqpublishnotifier.cpp
< sdaftuar>
maybe that's it though... if your changes were comprehensive for the RPC stuff (sorry I need to review still), then perhaps this isn't as bad as I feared
< instagibbs>
gettransaction also returns a hex field I'm not converting, but it feels weird messing with an api that is wallet-based for this purpose.
< instagibbs>
I think "raw" apis are a better fit for these changes. If you're using wallet features, just use wallet and what the wallet understands
< gmaxwell>
the overall goal should be "no one should need to hesitate to install 0.13.1+ simply because they haven't updated their applications yet". This was part of the attractiveness of having the segwit code in a release ahead of activation and avoiding behavior changes at activation time.
< kanzure>
should there be a nonce in rpc client requests? otherwise how does sendmany work if you had a socket timeout on the response? or other must-only-run-once critical rcp endpoints.
< kanzure>
'scuse me, *rpc endpoints
< gmaxwell>
kanzure: 'check listtransactions' :-/ you're right, it should have a nonce. Well the whole rpc thing was not well designed.
< kanzure>
alternatively, you could say "well, just call an event log, and check before you retry" (someone posted a "retry all rpc requests" to petertodd's python-bitcoinlib repo and it raised my eyebrow)
< kanzure>
if sending money is the only scenario where something bad happens, then listtransactions is probably sufficient...
< gmaxwell>
(or, better, a nonce, sequence number pair, and a nonce can be reused but only with a strictly greater sequence number)
< kanzure>
what about: in RPC request to sendmany (or whatever else is broken here), include expected state and also the actual request. and if the expected state doesn't match, then raise error or don't send.
< kanzure>
someone outside the channel proposes, 'if you want to enforce balance checking, make the server require the client to make a balance call before making a send call.'
< kanzure>
do these problems exist in the http api?
< sipa>
you mean the REST api? that's read only
< sipa>
and only for public data (nothing wallet related)
< sipa>
gcc 7.0 has -Wmisleading-indentation
< Chris_Stewart_5>
Just to be clear, there is no compact size uint to indicate how many CTxInWitness inside of the serialization of a tx right?
< Chris_Stewart_5>
You need to infer it from the number of inputs?
< bsm1175321>
Are there any RPC requests that would NOT be idempotent? I think kanzure's suggestion is that bitcoind always complete the action of an RPC call even in event of an I/O error while sending the response. Is it possible that this wouldn't be the case?
< bsm1175321>
Further, is it possible for a truncated RPC request to end up being valid and acted upon?
< bsm1175321>
BTW I think this is probably RPC "atomicity" not "idempotency".
< kanzure>
no, my suggestion is not "always complete the action of an RPC call even in event of an I/O error while sending the response" -- that's a related topic i guess.
< bsm1175321>
Nah kanzure's right in using idempotent -- CS lingo. Too much math over here.
< kanzure>
my suggestion is "change from 'always execute any RPC request' to 'check whether the nonce is the same first, and fail if the nonce is the same'"
< bsm1175321>
Sure. From the client's perspective the two situations are the same...he didn't get the response and doesn't know if the RPC sendmany was executed. So he retries with the same nonce...
< kanzure>
yes, retrying with the same nonce would be OK
< sipa>
an alternative is having meta support for any asynchronous command
< bsm1175321>
I like that idea...
< bsm1175321>
(I like kanzure's nonce idea). What's meta support?
< sipa>
you call an RPC "schedule" with 3 args: 1) command 2) command args 3) call id
< sipa>
and then you have a separate RPC through which you can chrck the result
< kanzure>
have you poked at database internals before, like transactional concept?
< sipa>
define 'poked'
< gmaxwell>
sipa: that kind of framework is _necessary_ for some commands.
< kanzure>
eh, 'poked' could be reading i guess :)
< gmaxwell>
For example automatic sendmany batching and replacement will not know the txid until later.
< kanzure>
daisychaining and 'schedule' sure. although if you require some of the prior information before generating the next call, ... can't send all at once.
< bsm1175321>
To be clear -- the issue we're having is that bitcoind closes its RPC connections on a timer. These are connections on localhost and python-bitcoinlib doesn't reopen them (I consider that it's bug). But this could also be improved on the bitcoind side if we could configure it to not close RPC connections...
< bsm1175321>
That's separate from calling RPC's over a flaky connection...
< kanzure>
rpc connection closing behavior is somewhat unrelated; your application still needs to know whether to retry. and currently it can't possibly know that.
< bsm1175321>
Yep yep
< gmaxwell>
"not close RPC connections." < guarentees file descriptor exhaustion. And is unlike any http server ever.
< gmaxwell>
bsm1175321: I think your concern is just a "fix the client" concern. (the python-bitcoinlib behavior has burned me many times too)
< bsm1175321>
But there's only one RPC user and it's on localhost. I think this is a common configuration.
< bsm1175321>
gmaxwell: I agree. I'm gonna fix it to re-open and not bother me with I/O exceptions.
< kanzure>
"only one rpc user" what?
< sipa>
didn't we fix that issue in our in-tree fork of python-bitcoinlib?
< gmaxwell>
often still results in exhaustion, e.g. when you get long idle connection objects that aren't GCed yet and so they've left their connections open.
< bsm1175321>
sipa: yes it looks like it is fixed there.
< skyraider>
connection handling is distinct from client state management. i think kanzure's concern is that it seems possible for a bitcoind client to say "hey, i didn't receive a reply" and then resend the same request to bitcoind - perhaps resulting in the accidental execution of two identical requests. banks used to behave this way when you clicked "do wire transfer"
< skyraider>
then pressed "back" in internet explorer - they'd do another POST to send the wire transfer again. now we have idempotency via request nonces for the web, so it's not a problem - your nonce is considered expired once the nonce's corresponding request is processed. with python-bitcoinlib in particular, i think the concern is that you can still get "oops i
< skyraider>
sent wire transfer twice" type behavior, due to no client state management facilities in bitcoind to code against.. is that right?
< gmaxwell>
skyraider: applications can check list transactions to determine if the order has been executed yet.
< sipa>
kanzure: that's just for ordering of responses within one connection
< gmaxwell>
A common model is to poll outstanding requests, check list transactions, anything not paid yet, pay... anything already paid remove from the list.
< kanzure>
gmaxwell: no, that doesn't work. listtransactions will not matter because you might have anohter concurrent attempt. then both sendmanys execute one after another.
< kanzure>
ok well at the very least we should insist that client librares implement that pattern, maybe by rejecting sendmany rpc requests if previous required rpc requests have not been detected
< gmaxwell>
"Don't do that."
< gmaxwell>
I'm just describing to you an architecture I've seen before, which has generally acceptable properties.
< gmaxwell>
I don't disagree that an optional nonce/sequence with replay protection would be good-- though it has some scalability challenges.
< kanzure>
sipa: oh like for ordering responses to the _batch call?
< sipa>
kanzure: you can send two rpcs with the same id, and you could get two different answers with the same id back
< sipa>
gmaxwell, kanzure: a more lightway idea, just for send* calls, is passing as an argument to the rpc the expected wallet balance
< sipa>
anything that actually interferes would cause failure, while other rpcs could still happen concurrently
< kanzure>
wallet balance isn't good either... many simultaneous deposits/debits could put you back to the same balance anyway.
< kanzure>
excuse me, credits/debits
< skyraider>
so 'read before you try to send lots of bitcoin' - alright. but if another RPC connection writes in the meantime, won't my read be stale, or is there transaction isolation somehow? is the recommended policy therefore 'only ever use a single RPC writer' ?
< gmaxwell>
sipa: ugh. so if you send some and recieve the same amount ... the double payment still goes through?
< kanzure>
'transaction isolation' probably means database-kind of transaction model, not transactions?
< skyraider>
uh, yeah, not bitcoin transaction. isolation as in the I from ACID.
< gmaxwell>
skyraider: yes, you just retry in that model.
< kanzure>
isn't that a failure
< kanzure>
the idea is to prevent stales from causing you to retry
< gmaxwell>
why would that be a failure? The retry causes no harm. The system is concurrent and you want an atomic change, there will always be retries needed.
< kanzure>
you mean existing system, or proposed system is concurrent?
< gmaxwell>
kanzure: so going back to the nonce, how do you propose avoding having to store nonces forever?
< skyraider>
gmaxwell: sorry, could you clarify please - is RPC safe for concurrent writes at the moment, or are you saying stale reads (if you have, say, two concurrent connections) are possible, and therefore one should use a single RPC writer?
< gmaxwell>
skyraider: your question is underspecified. It's perfectly fine to make concurrent writes. But kanzure is asking about what happens when you get a socket failure on one of your writes.
< gmaxwell>
If you just retry you may double pay.
< midnightmagic>
wtf man another earthquake near fukushima?
< skyraider>
right, i don't mean to ask whether the application will crash if you do concurrent writes. rather, whether a client can expect the situation of "1) read balance in client thread A, 2) write balance in client thread B that makes 1's read stale, 3) write balance in client thread A" can result in the client double paying, due to making a decision based upon the
< skyraider>
balance data in read 1).
< sipa>
skyraider: there is no 'write balance'
< sipa>
there is 'attempt to make a transaction'
< skyraider>
yes, sorry, shorthand for "try to send some bitcoin"
< gmaxwell>
skyraider: I think you're overthinking things now. If you call send* you will probably send. If you call send once per payment you want to make you will never double pay.
< gmaxwell>
if you are doing things based on 'balance' then god knows what will happen, as balance is not monotone. :)
< sipa>
skyraider: if you "attempt to make a transaction" twice, it will obviously send twice... anything else would be highly unexpected behaviour
< kanzure>
i think bitcoind seems to really very strongly want all applications to have only one rpc connection
< gmaxwell>
kanzure: not at all.
< kanzure>
there is no protection against stale reads at all
< kanzure>
there is nothing in here at all like ACID
< gmaxwell>
which is orthorgonal to "have only one rpc connection"
< kanzure>
look i'm fine with one rpc client in serial. that's fine with me. but i'm not sure everyone else knows to do that.
< skyraider>
it might help to put on a client's shoes here. context was https://github.com/petertodd/python-bitcoinlib/pull/128 in which a blind retry python decorator was considered. turns out this can a) result in actually sending bitcoin multiple times, b) result in decisions made on stale read data (balance was just an example - really any bitcoind state subject to
< skyraider>
change via concurrent writes). it seems a safe solution for b) is "only ever execute RPC writes serially" and for a), "because we are doing serial writes, in which another write can't occur after my safety-check-read, we can query to see whether the send already occurred (the safety-check-read) before retrying." tldr; serial writes enables "should i retry?"
< skyraider>
logic in a safe way.
< gmaxwell>
I disagree.
< gmaxwell>
What you want is a write which is ordered with respect to a read.
< gmaxwell>
Have you already paid is not something which can become untrue.
< kanzure>
previous answer to "have you already paid" can become untrue
< gmaxwell>
yes but only by performing a write that makes that payment.
< skyraider>
yes, a write ordered with respect to a particular read (i.e., transaction isolation in the database sense) is a more general and better solution than serial writes.
< gmaxwell>
Both of you keep repeating serial, and that is just wrong and in a meaningful way compared to typical usage. You can perform as many sends as you want. But if you want to retry a send you must read first to make sure the send wasn't already performed. Running two sends for the same payment in parallel is insanity in any case, as you have no way of knowing if the other went through (and usuall
< gmaxwell>
y it does).
< bsm1175321>
Of course, there are probably many people performing this logic now, since the "commit" of a send doesn't actually occur until it's mined. The need to fee-bump is probably a more common cause of retries than RPC connection drops...
< skyraider>
if you are e.g. running a group of ec2 servers or something and experience a temporary DNS failure of the kind that's common on AWS, that python-bitcoinlib decorator could result in probably 1-50 double payments per month, assuming you are sending every 30 seconds or so. serial writes seems to be a quick, safe way to get don't-pay-twice safety while querying
< skyraider>
the current version of bitcoind. as to running sends concurrently, yes, it seems unusual / nonrealistic use case to do this.
< kanzure>
fwiw i don't actually use sendmany
< kanzure>
or any spends for that matter... heh.
< sipa>
you hodler
< kanzure>
(yeah who would ever want to spend coins??)
< midnightmagic>
gmaxwell: that's how MP (operating manually) managed to rip himself off.
< kanzure>
midnightmagic: go on?
< skyraider>
and yeah i understand the criticism about serial writes - it seems at first it should be more like 'read before you write'. but, you have to be careful with that because your read could easily be stale. and if you are running concurrent bitcoind sends in different client-application-level thread of execution, well - you shouldn't do this. you should write
< skyraider>
in one thread because the lack of transaction isolation (read/write ordering) means client implementors might not know to handle what happens when a balance they just checked is now gone.
< skyraider>
but, yes, this is not an actual real-world use case over here at the moment. so admitted.
< skyraider>
thanks
< midnightmagic>
kanzure: not respending himself, instead sending multiple sends to the same destination more than once, and the receivers obviously refused to give anything back (which was fair by MP's own mercenrary rules.) If he had properly respent himself, updated his sends, or otherwise presumed the sends could be mined at any time, he wouldn't have lost anything.
< kanzure>
is the assumption that there is always a single node worker and never more than one? if so, we should probably add this to docs.
< kanzure>
*node worker (rpc client communicating to bitcoind)
< bsm1175321>
Having more than one violates the Isolation requirement of ACID. Would it be interesting to require isolation on open wallets WRT rpc clients? That is, two RPC connections must have mutually exclusive wallets?
< gmaxwell>
kanzure: I keep telling you that there is no such assumption.
< sipa>
kanzure: that also doesn't in any way help with the double issue problem