< gmaxwell>
sipa: you broke the sse4 thing by renaming incompletely. fixy fixy.
< sipa>
gmaxwell: fixed
< cfields>
jeez, qt 5.9 made a mess of _everything_
< gmaxwell>
What did they do
< cfields>
they're working on modularizing their source/build. but the result (so far) is just a new tangled mess of dependencies. And no working configs without patches
< cfields>
want cocoa for osx? you'll need a printer of course. And obviously printers require opengl...
< luke-jr>
O.o
< sipa>
any reason why we'd really want 5.9?
< sipa>
cfields: they're 3D printers perhaps? :D
< cfields>
sipa: lol
< cfields>
sipa: ironically i was looking forward to 5.9 and its slimmer build
< gmaxwell>
they probably need opengl for 'lpad()'...
< sipa>
or for is_integer
< cfields>
but after messing with it all day, it's not offering many advantages so far
< cfields>
(other than providing me a stack of patches to upstream)
< cfields>
so no, i don't think it makes sense to bump
< cfields>
as-is, we don't build with 5.9. But I haven't seen a bug report yet, so I'm assuming distros aren't shipping it
< cfields>
(and it'll be a pain to support once they do :( )
< luke-jr>
Qt used to care about backward compat :/
< luke-jr>
I noticed some stuff I have won't build with 5.7 because it requires C++11 >_<
< cfields>
luke-jr: i suspect all is fine if you use qmake
< luke-jr>
ah
< cfields>
yea, c++11 became a hard dep for 5.7 i believe
< luke-jr>
well, FWIW, Gentoo doesn't have 5.8 as even an option yet
< cfields>
but stl is still optional i think :p
< luke-jr>
hard dep makes sense; but 5.7 won't allow Qt programs to use non-C++11 :p
< sipa>
gcc 6 by default already compiles with c++14
< luke-jr>
anyhow, so I changed my USE flags to build that app for Qt4 <.<
< cfields>
mm, that's annoying. but kinda makes sense from a support standpoint
< wumpus>
any reason to not just stick with our current qt for 0.15? that sounds like a nightmare, better let them sort out their problems upstream and not be a guinea pig
< wumpus>
from a user PoV there seems to be no difference between different qt version for a long time now, as we basically only use the 4.x API
< luke-jr>
they don't even have a Qt5.9 dance, so..
< wumpus>
hehe
< wumpus>
disabling -getinfo on the day of the feature freeze, really?
< luke-jr>
meh, why not. there's an option to get it back
< wumpus>
I don't get it, what is the sudden rush?
< wumpus>
sure, but we could have done that 100 times during 0.15's release cycle
< wumpus>
but somehow it has to be added in the last day
< gmaxwell>
leeroy jenkins!
< luke-jr>
I suppose it adds a new string
< luke-jr>
(for the -help)
< gmaxwell>
I think it should be done at the beginning of a cycle. It's bound to cause severe irritation for people who use it at the CLI and we will predictably get PRs to improve other things to compensate.
< wumpus>
gmaxwell: heh indeed, in this way we don't even get used to it ourself
< wumpus>
before exposing it to end users
< wumpus>
can't we just add a comment to the getinfo output instead?
< wumpus>
Warning: this will be deprecated!
< gmaxwell>
(or perhaps I'm alone in continuing to use getinfo myself? :) )
< wumpus>
instead of the add a command line argument dance
< gmaxwell>
"nag": "This RPC will be deprecated."
< wumpus>
gmaxwell: I officially stopped using it at least, replaced it with a client-side script that shows more useful information
< luke-jr>
wumpus: +1, maybe even put it in the warnings software already should look at?
< wumpus>
getinfo's output is pretty meh, do you ever lok at anything besides the number of connections or the wallet balance?
< wumpus>
luke-jr: not sure that's a good idea
< wumpus>
luke-jr: it's not part of the node-wide warnings
< luke-jr>
not sure why that'd be a reason not to do it
< wumpus>
because it's a false alarm
< wumpus>
it's not actually an alarm
< luke-jr>
"your software is about to break" isn't? ;)
< gmaxwell>
Connections, block count, balance, warings. And unfortunately, to use seperate rpcs to get this info is three rpcs (four?) and most of them each give a full screen full of information each. :)
< luke-jr>
getblockcount and getbalance are scalars
< sipa>
is there any rpc besides getinfo that actually lists the warning?
< wumpus>
what the hell
< wumpus>
BlueMatt concerned about atomicity in #8843?!
< gmaxwell>
true, though I was using getblockchaininfo and getwalletinfo as replacements.
< wumpus>
never mind, going to put this discussion on the backburner, there's more important things to worry about
< wumpus>
what needs to get in today?
< luke-jr>
BIP148, but I assume if the hold-outs on that had changed their mind, they'd say so :/ so probably nothing to do / discuss there
< wumpus>
no no no no no
< sipa>
#10831 would be nice, together with #10830 (which i hope can go in s a bugfix later)
< gribble>
https://github.com/bitcoin/bitcoin/issues/10831 | Batch flushing operations to the walletdb during top up and increase keypool size. by gmaxwell · Pull Request #10831 · bitcoin/bitcoin · GitHub
< gmaxwell>
so on #10831 I'm ambivlent to the performance improvement; I created the PR because I understood that performance issue to be a blocker for increasing the default keypool size, which I think is really important to get out due to bad interactions with hd wallet and rescan.
< gribble>
https://github.com/bitcoin/bitcoin/issues/10831 | Batch flushing operations to the walletdb during top up and increase keypool size. by gmaxwell · Pull Request #10831 · bitcoin/bitcoin · GitHub
< luke-jr>
we should probably just do keypool in the background..?
< bitcoin-git>
[bitcoin] practicalswift closed pull request #10847: Enable devirtualization opportunities by using the final specifier (C++11) (master...devirtualization) https://github.com/bitcoin/bitcoin/pull/10847
< wumpus>
well, performance improvement or not, doing the keypool top-up in a single db transaction makes sense
< luke-jr>
doing it in a single dbtx would necessitate blocking on it, though, no?
< sipa>
?
< luke-jr>
at least before any of the keys can be used
< wumpus>
especially on VMs with slow sync (most of them), it's super slow as it is right now
< wumpus>
blocking on *what*? key generation is so fast
< sipa>
if it weren't for db syncing, we can generate 10000 per second or so
< wumpus>
the only reason it takes noticable time right now is because of the sync per key
< gmaxwell>
luke-jr: the topup is already a single operation that holds the relevant locks the whole time.
< luke-jr>
i c
< luke-jr>
gmaxwell: yes, I was thinking we shouldn't do that, but otoh, if this is so much of a gain, maybe it doesn't matter
< wumpus>
it's only done once anyhow!
< gmaxwell>
sipa: well not quite that fast, the code paths it goes through per key are very twisty and inefficient, and we aren't multithreaded for it, and we do validate after create. :)
< wumpus>
(at least, so much at once)
< gmaxwell>
but thousands per second sure.
< wumpus>
keypool generation happens a lot during testing though, so #10831 is going to speed up testing
< gribble>
https://github.com/bitcoin/bitcoin/issues/10831 | Batch flushing operations to the walletdb during top up and increase keypool size. by gmaxwell · Pull Request #10831 · bitcoin/bitcoin · GitHub
< wumpus>
(euh.. it would at least if if the number of keys generated stayed the same)
< gmaxwell>
luke-jr: if someone really cared, they could make topup run in batches of no more than X to reduce worst case blocking, it would be a couple lines improvement probably.
< gmaxwell>
wumpus: it'll be faster even with the 1000 size increase, esp on things with slow IO.
< wumpus>
ah yes it's 1000, yes srue
< wumpus>
does it still log a line for every key?
< wumpus>
probably want to get rid of that too
< luke-jr>
well, if it's so fast, it doesn't matter
< gmaxwell>
yes.
< luke-jr>
"added N keys to keypool" ☺
< gmaxwell>
instagibbs reported for 1000 keys (new default) on his presumably SSD equipmt system it went from 20 second to 1.6. so I expect it's still a 20% speedup. We could in the future make many of the tests turn their keypool sizes down to speed them up further.
< gmaxwell>
it would be a trivial (2 loc-ish) change to make it log only one line in the topup.
< wumpus>
should certainly do that, I've always found the per-key message annoying during first run, and it's 10 times as annoying now :)
< gmaxwell>
as in, remove the current line, re-add it two below using the missingInternal missingExternal variables as the count.
< gmaxwell>
okay, should I just add another commit that does that to my PR?
< sipa>
ack
< wumpus>
remove the current line, or move it to debug category if someone theoretically could be interested while troubleshooting
< gmaxwell>
I can't see the reason, it conveys no useful information that the summary wouldn't have.
< wumpus>
agree
< gmaxwell>
and yes, the old one annoyed me too. though leveldb logging with debug on has made me blind to log annoyances... (funny, I did that nice bitfield thing and I don't use it to deactivate leveldb)
< wumpus>
heh yes that's kind of what it was meant for
< wumpus>
though now that it's possible to enable/disable debug bits at runtime, I don't enable any by default anymore
< wumpus>
well, bench. That one is nice.
< gmaxwell>
pushed, not yet tested because it'll take me a half hour to compile it.
< wumpus>
I can test np
< gmaxwell>
I will test, just... when the build finishes.
< wumpus>
half an hour for a single-line change, did you change to rpi for builds? :-)
< gmaxwell>
wumpus: my laptop is slow, and I've switched trees, so its recompiling all the things.
< gmaxwell>
yea, thats still pretty slow considering how fast the underlying crypto is... but fast enough.
< wumpus>
well it's short enough not to be noticable in the overall process, could always be optimzied further if anyone cares
< gmaxwell>
if we later change it to 10k it might be worth it.
< wumpus>
184s for 10k keys here, yea that's definitely too slow
< wumpus>
sorry - 100k keys
< wumpus>
but 18.4 seconds would also be too slow
< gmaxwell>
it could also be broken up and run in the background, I didn't push for 10k for a different reason: bloats up the wallet currently.
< gmaxwell>
esp if you use encryption and immediately mark all the keys you just generated as used.
< wumpus>
then again you had a good point about the encryption
< wumpus>
yes
< gmaxwell>
but we can fix this later by not even creating the wallet until you do something like hit getnewaddress or do encryption.
< wumpus>
there could be some kind of shortcut - if you *never* dealt out a key before encrypting
< wumpus>
then it could just start over with an encrypted wallet
< gmaxwell>
or that.. and also, by using smaller records for keypool entries.
< wumpus>
for multiwallet it'd also be nice to be able to create wallets in encrypted mode
< wumpus>
yes
< gmaxwell>
I think we should generally consider defered init of the wallet because it would allow us to better integrate mandatory backup/recovery.
< gmaxwell>
e.g. in the GUI at least you don't do a wallet unless you complete a series of slightly paternalistic steps that make it likely you actually got some kind of backup.
< gmaxwell>
so: starting with no wallet, and any effort to get an address prompts you to make one, along with a backup of some form.
< morcos>
wumpus thanks for merging 10706. The last thing I want to nag about is #10707. It hasn't gotten much review b/c it was dependent on 10706, but its not big. I think its important in terms of finalizing estimatesmartfee API now that it is no longer marked unstable. Not sure if you feel the cutoff for it is today.
< morcos>
#10817 would be nice (and isn't yet tagged 0.15) but isn't critical, and so far I haven't heard anyone other than gmaxwell and I's opinion on it.
< bitcoin-git>
[bitcoin] jonasschnelli opened pull request #10849: Multiwallet: simplest endpoint support (master...2017/07/mw_endpoint_simple) https://github.com/bitcoin/bitcoin/pull/10849
< gmaxwell>
morcos: I would not for 10817 that the definition used there is a bit different then the one were thinking of using for the exact matching hurestic, because the dust threshold has a *3 in it.
< gmaxwell>
I think thats okay, but its slightly less of a no-brainer than the factor of one hurestic (where the output is worthless at the long term feerate).
< morcos>
gmaxwell: Yeah I argued a release ago we should remove the *3 from the definition of dust and correspondingly change the dust rate to 3 sat/B . It would make more sense to talk about everything like that.
< gmaxwell>
morcos: I would agree with that change, makes a lot of sense.
< gmaxwell>
if you make a commit that does that, fix the comment in IsDust which just seems to be inexplicable.
< gmaxwell>
(there is no integer you can multiply 148 by to get 546)
< morcos>
still want to try to do that for 0.15? or forget it for now
< morcos>
heh!
< gmaxwell>
Well I think you should put it in 10817, other than the 10817 behavior it's a no-op
< morcos>
not forget, but postpone
< morcos>
ok
< gmaxwell>
I'd like to get it in 0.15, but wladimir's call. I think the PR would be better with it, esp as it makes it even a more benign change.
< gmaxwell>
(also, I don't really think we have any other utxo bloat reducers in this release; gotta meet quota)
< morcos>
But then you would make the discard rate still 5 sat/B or you'd make it higher
< gmaxwell>
morcos: the 3x in there was due to the original somewhat stupid formulation that tied it to minimum relay fee.
< morcos>
in the PR I gave the long term fee estimation rate can only serve to LOWER the discard rate
< morcos>
it is not a max
< morcos>
I think it is dangerous to use the existing long term fee estimation rate as a discard rate
< morcos>
it got as high as 100 sat/B a couple months ago
< gmaxwell>
Could be argued that leaving it at 5 is just more conservative and better than not having the feature.
< gmaxwell>
5 means the threshold is about at 1.5 cents at the current prices.
< morcos>
it seems not worth the complication to increase the discard rate from the preexisting 3 sat/B b/c of dust to 5
< morcos>
from 3 to 15 then yes
< morcos>
and i think the fact that it gets MIN'ed with long term fee estimation makes me not worry too much that the 15 will end up biting people in the ass later. (also it's configurable)
< gmaxwell>
Yes, that also sounds fine to me.
< morcos>
only other question is it's a bit dicey to change the definition of a command line argument like dustrelayfee, but in this case it's a hidden argument that i doubt anyone specified, so its probably ok.
< morcos>
but thats more reason to do it now while it's only been out for one release
< gmaxwell>
it's also good to get this in because the exact matcher code should signficantly increase the number of cases where this happens, so if people are gonna squak about the wallet throwing away a few more cents to avoid change it'll be good to hear about it.
< morcos>
gmaxwell: 546 = 3 * (148 + 34)
< gmaxwell>
ah thats where it comes from; it's including the txout itself.
< wumpus>
happy that we're moving to an API where all the important things can be set per transaction, so that different callers don't have to worry about messing up each other's global state
< wumpus>
only 3 things in project 8 anyhow - maybe we should clean it out for now, and use the 0.15 milestone list instead
< wumpus>
(it's what I'm doing at least)
< wumpus>
(one thing now)
< jonasschnelli>
CodeShark: have you started to work on client side filtering after roasbeef's specs?
< jnewbery>
Chris_Stewart_5: to get all blocks and transactions to propagate, try BitcoinTestFramework.sync_all()
< jnewbery>
wumpus gmaxwell: for getinfo, "nag": "This RPC will be deprecated." will be silently dropped by any clients that are picking individual fields from getinfo.
< jnewbery>
Hiding getinfo behind a flag for one version *forces* users to acknowledge that the RPC is being deprecated (and gives them one release grace period to fix their scripts/whatever that is using getinfo)
< jnewbery>
wumpus: > keypool generation happens a lot during testing though
< wumpus>
jnewbery: didn't know that the keypool size was overridden for testing, so in that case this doesn't help testing become faster, well too bad :)
< wumpus>
jnewbery: yes, we can do the other thing for 0.16, just too late to make such a change for 0.15
< wumpus>
jnewbery: I honestly don't understand why this comes up on the last day before feature freeze
< jnewbery>
I agree, it's late. But hiding getinfo behind a flag in v0.16 => removing in 0.17. getinfo is one of the bitcoin_server -> bitcoin_wallet dependencies. I was hoping we'd be able to remove all of those for 0.16
< wumpus>
sure, but couldn't you have brought it up a few weeks earlier?
< gmaxwell>
I'd still complain that we haven't dogfooded it enough.
< wumpus>
we could just remove the wallet balance from there
< gmaxwell>
I wouldn't complain about that.
< wumpus>
it makes no sense with multiwallet anyway
< wumpus>
then again, that doesn't need to be done last-minute for 0.15, the circular dependency won't be solved before then anyhow
< wumpus>
but let's say it's the last thing holding back solving the build dependency, we could do that
< wumpus>
I still think adding the nag: makes sense, most people using getinfo will likely use it from the command line
< gmaxwell>
even if they don't use it always that way, it's more visible than even release notes.
< wumpus>
yes it would be in addition to release notes, not instead
< gmaxwell>
and we've mentioned it in release notes before (I think, I hope)
< wumpus>
yes, we did
< wumpus>
even with the new places where everything could be found
< wumpus>
I don't know why we waited for 0.14 before doing that
< BlueMatt>
yeesh y'all been busy this morning
< wumpus>
anyhow as it was deprecated in 0.14 (the deprecation message in the help was even backported to 0.13.1), removing it completely in 0.16 should be fine
< BlueMatt>
wumpus: I mean its not a strong concern, really, I pointed it out cause it would be a problem for automated users (which I hope are mostly not calling -cli)
< wumpus>
BlueMatt: yes, the -cli change would mostly be for command line users, but now agree we shouldn't do that, external stats/info scripts should be external
< BlueMatt>
well i was more curious to find out what the demand for it was (re: the "IRC discussion" referenced in that pr)
< wumpus>
and -cli should be a minimal, dumb client utiltiy
< gmaxwell>
I think we'll have to do ~something~ but the best way to find that and get it done is just to live with it for a while.
< wumpus>
maybe some example of a stats script? I don't know - generally, those things have gone unmaintained fast, as they're personal, everyone is interested in different info
< wumpus>
there is also such a low threshold to writing them
< gmaxwell>
connection count + height + errors is pretty universal; and is a bit obnoxious from the cli in our post getinfo world.
< jnewbery>
> couldn't you have brought it up a few weeks earlier?
< wumpus>
well for example personally I'm not interested in totai connection count - my script shows in/out versus ipv4/ipv6/tor
< jnewbery>
I brought it up in response to #10841. I thought it would be friendlier to hide the RPC behind a flag before removing it one release later
< gmaxwell>
wumpus: the normal thing I'm interested in is 0/8/many. seperate in/out counts would be interesting. From a cli perspective there are a number of other interesting things to show; e.g. best block hash, without getting into those huge info dumps like getblockchaininfo.
< * BlueMatt>
has 7 4k monitors full of stats monitoring scripts...anyone need some?
< wumpus>
jnewbery: sure! BlueMatt was the original wtf 'why do you bring this up now', you tried to do a friendlier version, which is why I tagged that one
< jnewbery>
If we're going to remove getinfo in v0.16, I suggest I close #10841
< wumpus>
jnewbery: but I stil think it is too much for such a late change
< wumpus>
gmaxwell: exactly, none of which getinfo currently does, it's been frozen for years (on purpose)
< * BlueMatt>
had assumed it had been longer-deprecated, so figured it could just be removed, when it was pointed out that it was only deprecated in 14 (damn it) i withdrew, but making the deprecation more explicit is nice
< BlueMatt>
i guess it doesnt matter much, though
< jnewbery>
Yeah, doesn't matter too much. I'll close 10841.
< wumpus>
we should have mentioned it in the release notes much earlier
< wumpus>
jnewbery: I still think it makes sense to add a 'nag': to the result with your message, but anyhow
< gmaxwell>
wumpus: I would have log ago proposed a 'status' rpc, but feared that the response would be "omg not another getinfo!"
< wumpus>
gmaxwell: yes we're definitely not going to have any new server side call like that, that spans subsystems
< jnewbery>
yeah, adding a "nag" field does no harm
< wumpus>
client-side would have been ok with me, but runs into too much opposition, so we';re not going to do that either
< bitcoin-git>
[bitcoin] jnewbery closed pull request #10841: [rpc] Give users one final warning before removing getinfo (master...deprecate_getinfo) https://github.com/bitcoin/bitcoin/pull/10841
< gmaxwell>
wumpus: if it didn't span into the wallet, but was just blockchain stats and p2p stats?
< wumpus>
gmaxwell: I don't think that's a good idea, no
< gmaxwell>
I figured.
< wumpus>
could potentially create another dependency in the future that will be so hard to get rid of
< wumpus>
if getting rid of getinfo would have been easier, I would have thought differently about it, but this keeps haunting us forever
< gmaxwell>
well, we have to do something. I don't pretend to know what. But having to run multiple commands just to get the most basic of health information, is a real burden for support.
< BlueMatt>
can we call #10526 a fix for a performance regression (really, disk space usage regression) and tag it 15 and merge it later this week?
< wumpus>
then do something: write a useful script taht collects that information, submit it as PR
< wumpus>
I TRIED
< gmaxwell>
I think we haven't gotten rid of it because we haven't replaced it for anything but programmatic users who care little about how many calls they make or how many pages of uninteresting data they toss.
< wumpus>
I just mistakingly made it part of -cli
< gmaxwell>
wumpus: I know. Though I admit I have mixed feelings about cli being anything but a dumb client too. I am not yelling at you.
< gmaxwell>
you did great.
< gmaxwell>
BlueMatt: I believe we were waiting on feedback from wumpus to determine if there actually was a need there.
< wumpus>
things that don't have to be done server side don't need to be done server side, this is the same reason I argue against #10804
< wumpus>
client side has the information, so can roll the statistics and display them in any way
< gmaxwell>
yes, I also saw 10804 and thought that.
< wumpus>
the RPC interface doesn't have to acoomodate end users specifics
< gmaxwell>
then our issue is that we just don't have a cli interface, except we have a highly used cli interface.
< gmaxwell>
:)
< gmaxwell>
BlueMatt: there was some question about the state of the node in question where he saw the not good results.
< wumpus>
the same problem we had in 2012, and have had extensive discussions about since then, and no one is making that tool it seems
< wumpus>
which I understand, I'm certainly not going to do it, too many opinions about it, I only write simple utility scripts for myself :)
< BlueMatt>
wumpus: hmm, ok, now I understand the desire, thanks, yea thanks for making that pr. would be nice to just make it a bash script, but sadly fucking json :(
< wumpus>
BlueMatt: it's trivial in python though
< gmaxwell>
rename bitcoin-cli bitcoin-rpc and we create a real bitcoin-cli ? :P
< BlueMatt>
true
< BlueMatt>
lol
< mmgen>
Just adding my two bits here: how about creating an addition cli command that aggregates info from various RPC calls and presents it in a readable way?
< wumpus>
I have that script, could post it, but don't really feel like submitting such trivial things to the repo
< wumpus>
mmgen: I DID THAT
< mmgen>
wumpus: what's it called?
< wumpus>
mmgen: no one liked it!
< BlueMatt>
lol
< mmgen>
I think that's the best solution
< mmgen>
It could even be coded in Python
< wumpus>
mmgen: I have a python and c++ implementation!
< gmaxwell>
wumpus: I liked it except for one concern, the parity between the CLI and RPC is a major learing curve improvement. If we went down the path of a highly cooked bitcoin-cli then knoweldge wouldn't automatically transfer between them, improvements wouldn't automatically transfer, etc.
< mmgen>
wumpus: need to convince the others I think
< wumpus>
I don't have the python one online, but could post it if anyone cares
< * BlueMatt>
was not aware of the python version
< BlueMatt>
yea, I'd be interested in putting the python one in contrib
< gmaxwell>
for a getinfo replacement that cooked vs not cooked argument doesn't really hold, since it would just be one command.
< wumpus>
gmaxwell: sure, that's just a matter of documentation though
< gmaxwell>
which could be documented ('here are the underlying rpcs')
< wumpus>
gmaxwell: it could show where it collects the various fields from for getinfo
< wumpus>
yeah...
< mmgen>
My wallet basically does just that, so I have a bit of experience in this area
< wumpus>
a python script is self documentingi nthat regard
< wumpus>
and easier to edit if you want something else
< wumpus>
so ok, I'll add that to contrib, not in 0.15 though
< gmaxwell>
wumpus: if it went as far as .e.g the ncurses interface, then that isn't the case. :)
< gmaxwell>
wumpus: we could use more python rpc examples, IMO, in any case.
< wumpus>
gmaxwell: yes I mean a simple getinfo replacement
< wumpus>
ncurses isn't trivial (though also not rocket science, I've gotten people to start linux coding using it)
< promag>
I'm happy to close #10804.. the problem is that to compute the histogram it takes really long to get and dump the json to the client with large utxo set
< mmgen>
wumpus: ncurses is for wimps. real programmers use ANSI escape codes
< wumpus>
mmgen: HAHA ansi escape codes are easier imo
< mmgen>
wumpus: the fact is they are
< BlueMatt>
morcos: wait, i thought min conftarget was 2?
< wumpus>
(depending on what you want to do, if you need windowing and partial refresh etc, and menus, ncurses is definitely the way to go)
< mmgen>
wumpus: true
< gmaxwell>
promag: how long is really long? for how large? With thousands of txouts it still takes a fraction of a second to listunspent last I checked.
< wumpus>
if you want to go wild with displaying 24-bit color unicode graphs, ansi escape colors are definitely easier
< wumpus>
promag: sure, computing the statistics server-side will be faster, but it's all a compromise...
< wumpus>
promag: in that case maintaining a private patch set might make sense - the use for those statistics is probably specific to you
< gmaxwell>
there are certantly things where I've written a python script to collect some data.. started it.... got tired of waiting, wrote it into the daemon on another host, recompiled, ran it, got the result...
< promag>
gmaxwell: let me get fresh numbers for you
< gmaxwell>
its unfortunate that rpc from python (and perhaps rpc in general) is so slow, though it is much faster than it used to be.
< BlueMatt>
yea, Ive run python rpc clients that literally look days to complete
< mmgen>
gmaxwell: I use it all the time. Haven't noticed that it's slow
< mmgen>
but this is with my own RPC library
< wumpus>
in that case it makes sense to implement it server-side in c++, but less sense for it to be merged upstream
< gmaxwell>
mmgen: try doing something simple like graphing the size of all blocks from it... takes rather long time. (or better, fee income from each block) at least with the normal python bitcoinrpc stuff
< wumpus>
there's not really any API that can help you in that case, I guess there could be a dynlib plugin interface to load things into bitcoind, but does that ever make things easier than just recompiling?
< mmgen>
gmaxwell: I don't use python-bitcoinrpc. I just connect to the daemon directly
< gmaxwell>
I believe we've factored up the relevant interfaces now that it shouldn't be hard to maintain external patches for things like this. Presumably we could take other changes that improved it further.
< gmaxwell>
mmgen: may be that it's mostly the python side handling making it slow now. (didn't used to be the case, but the rpc has become a lot faster)
< mmgen>
gmaxwell: Probably is. Python is slow
< mmgen>
in general
< morcos>
BlueMatt: huh?
< BlueMatt>
morcos: new docs for estimatesmartfee says nblocks minimum is 1?
< BlueMatt>
in #10707
< wumpus>
sometimes speeding up python is easy, a lot of code works as-is with pypy
< morcos>
it is the minimum you can request, however asking for 1 will give you an answer for 2
< mmgen>
wumpus: but everything remains single-threaded anyway
< BlueMatt>
morcos: yes, sorry, was reviewing the first commit first :p
< BlueMatt>
morcos: so lets just make the minimum 1?
< BlueMatt>
(or at least mention that in the docs)
< morcos>
it always worked to ask for 1, i didn't want to change that
< morcos>
it seems like a semi-common thing a user might ask for
< BlueMatt>
ok, so mention it in the docs, i guess
< wumpus>
mmgen: it's still quite a big linear speedup for tight loops, better than you'd get with multithreading in most cases
< wumpus>
(and it's no extra work, unlike multithreading)
< morcos>
Also I think it preserves the ability to in the future return something different/smarter for a as fast as possible conf_target of 1
< morcos>
BlueMatt: ok, while I'm at it, should I change the return parameter name from "blocks" now that we changed the argument to "conf_target" or leave it?
< bitcoin-git>
bitcoin/master dba485d Wladimir J. van der Laan: init: Factor out AppInitLockDataDirectory...
< bitcoin-git>
bitcoin/master 89bb036 Wladimir J. van der Laan: Merge #10832: init: Factor out AppInitLockDataDirectory and fix startup core dump issue...
< bitcoin-git>
[bitcoin] laanwj closed pull request #10832: init: Factor out AppInitLockDataDirectory and fix startup core dump issue (master...2017_07_appinitlockdatadirectory) https://github.com/bitcoin/bitcoin/pull/10832
< bitcoin-git>
[bitcoin] laanwj closed pull request #10831: Batch flushing operations to the walletdb during top up and increase keypool size. (master...topup_batch_flush) https://github.com/bitcoin/bitcoin/pull/10831
< cfields>
wumpus: re qt, no, i don't think it's worth messing with for now
< wumpus>
ok
< BlueMatt>
gah, github is missing notification emails :(
< BlueMatt>
wumpus: hmmm....I think (in addition to github actually missing notification emails) github wont send an email for a merge unless you comment merged.
< BlueMatt>
can you do that?
< wumpus>
hm?
< morcos>
BlueMatt: I've never gotten more than about 1 in 3 of my expected github notification emails
< BlueMatt>
eg #10831
< gribble>
https://github.com/bitcoin/bitcoin/issues/10831 | Batch flushing operations to the walletdb during top up and increase keypool size. by gmaxwell · Pull Request #10831 · bitcoin/bitcoin · GitHub
< BlueMatt>
morcos: did you mail support@?
< wumpus>
github should send an automated mail when something gets merged
< BlueMatt>
morcos: i just for the first time missed some emails
< morcos>
i complained to sdaftuar, does that count
< BlueMatt>
........
< morcos>
it didn't used to happen to him, but now it does too
< BlueMatt>
well whatever, I have all mailserver logs and didnt get any, so I'm gonna bitch to support@
< wumpus>
but I don't know what you mean with 'comment merged'
< BlueMatt>
wumpus: as in usually when sipa merges he comments on the pr merged when he closes it
< wumpus>
being the last things he merged, he commented on none of them, besides an utACK in some cases, but no "Merged". Github does that.
< BlueMatt>
ugh, yea, so github notifications ar ejust fucked
< BlueMatt>
yea, ok, ill go bitch at them
< BlueMatt>
clearly we should be doing dev on our own hosted infrastructure so that we dont miss notification emails :p
< wumpus>
as if sending mails from your own infrastructure (and having them actually arrive) is a sure thing, or was that the joke? :)
< BlueMatt>
it was a joke
< gmaxwell>
"now you have two problems"
< BlueMatt>
yea
< BlueMatt>
i mean they'd arrive for me, cause i run the receiving mailserver too :p
< gmaxwell>
BlueMatt: but feel free to setup mirrored infra... :)
< BlueMatt>
lol, I'll pass
< BlueMatt>
i already manage too much shit
< gmaxwell>
What do we think of #10817 ? I didn't notice until a day ago that it wasn't 15 flagged, I think it's ready to go. And its a likely useful lead up to work in 0.16-- in that it will increase change discarding somewhat, while planned work for 0.16 (the exact match coin selecter) will increase it a lot more... might be good to have in field feedback (are people going to be irate about it throwing
< morcos>
As for multiwallet, everyone just vote 0, 1, or 2 votes for either #10829 or #10849 based on how stronly you feel, and lets just pick one. honestly, we'll be fine with either choice, but lets concentrate our effort on one of them
< jnewbery>
but if other people have reviewed and tested #10829 then I can also live with that. I see the multiwallet interface in v0.15 as unstable/experimental, so as long as something gets in, I'm happy
< instagibbs>
I mostly reviewed that things behaved like they should, I can do the same today for either of the other two. I'm happy with whatever works.
< gmaxwell>
-1 on 10650, I think like 10849 better than 10829, as I think it will be more usable in practice but I will be fully willing to help throw out either interface later, so I don't care overly much. 10849 is a newer pr... both presumably still need a lot of review love.
< gmaxwell>
morcos: care to review #10672
< gribble>
https://github.com/bitcoin/bitcoin/issues/10672 | Avoid division by zero in the case of a corrupt estimates file by practicalswift · Pull Request #10672 · bitcoin/bitcoin · GitHub
< cfields>
BlueMatt: are you still wanting #10652 for 0.15 ?
< cfields>
gmaxwell: thanks for the reminder. fixing that up now.
< BlueMatt>
fizzwont: nah, think its gonna slip now
< BlueMatt>
cfields: ...
< instagibbs>
little worried about 10829 since named support is still a bit flakey. I think merging that would mean fixing up named arg support becomes priority.
< BlueMatt>
it would've been nice, but i dont think its critical enough to care, just untag 10672
< gmaxwell>
BlueMatt: whats the 'fizzwont' context for your message?
< BlueMatt>
gmaxwell: bad auto-tab, its a user on here
< cfields>
:(
< BlueMatt>
cfields: meh, its mostly a bugfix for not-really-issues bugs
< BlueMatt>
the cleanup to do the demangle is gonna be a through-16 process
< fizzwont>
i'm honored...but just observing
< bitcoin-git>
[bitcoin] gmaxwell opened pull request #10854: Avoid using sizes on non-fixed-width types to derive protocol constants. (master...rbf-numlimit-fix) https://github.com/bitcoin/bitcoin/pull/10854
< morcos>
instagibbs: to be clear, there are no rpc calls where you can't use named arguments right? there just might be some where you need to specifically specify the default value if you want to name a later argument (but this is the same thing you have to do with positional)
< sipa>
wasnt't there a PR to fix that?
< instagibbs>
morcos, oh, true
< instagibbs>
yes you'll have to enter in all values for some
< morcos>
sipa: yeah, i was just trying to understand the urgency
< instagibbs>
merely annoying
< BlueMatt>
lol, somehow i thought 10758 was already tagged 15
< morcos>
i guess people didn't like my voting idea or don't have an opinon on which multiwallet PR to merge. can we decide so we can make it happen? i'm happy to review 10849 if we prefer that one
< sipa>
i'll review the last PR before giving an opinion
< ryanofsky>
i'd give one vote to 10829 (obvs), and one conditional vote to 10849 if it uses a plain query parameter, or has some documentation explaining the path schema
< sipa>
i think a query parameter is inferior to just a named parameter, and doesn't really simplify later moving to a new process
< sipa>
i'm fine with named parameters for now, as a stopgap measure because a full new API isn't complete
< sipa>
documentation can come later
< sipa>
that doesn't need to be ready by the feature freeze
< sipa>
but i do think a new endpoint is eventually the way to go - whether that happens in 0.15 or not
< ryanofsky>
sipa i think moving to new process is basically orthogonal. reason why i think query parameter is better is it leave uri-path space open for future uses and an actual thoughtful design. current design is changing multiple times a day
< ryanofsky>
add v1, subtract v1, forward node calls through wallet, then stop, then start again
< sipa>
ryanofsky: i really disagree with that... the point of an endpoint is that clients can be configured to point to a new process
< ryanofsky>
i'm trying to understand...
< sipa>
you can't do that with query parameters
< ryanofsky>
? why not?
< sipa>
what if some of them are process-specific, and some or not?
< sipa>
the client software would need to go modify the url to put in extra things
< sipa>
i guess it isn't worse - as long as everything you'd put in the path becomes a query parameter and nothing else
< sipa>
but it also doesn't gain you anything
< jtimon>
ryanofsky: it's about simplyfing concurrency
< sipa>
jtimon: wut?
< jtimon>
is it not?
< sipa>
what does that have to do with anything
< ryanofsky>
i'm not understanding what you're saying sipa... did you change your mind midway?
< sipa>
ryanofsky: yes, i did; i agree that query parameters can do anything that a URI can... but it also doesn't gain you anything
< ryanofsky>
advantage of query parameter is it leaves more options for the future that don't require us to break compatibility
< ryanofsky>
nothing to do with multiprocess
< ryanofsky>
specific example of what i'm talking about: maybe we want to have web interface listening on /wallet. maybe we want to add versioning in the future
< sipa>
ryanofsky: but the point is creating a namespace
< jtimon>
right, each wallet can get its own lock even with query paramters, then I don't undesrtand what's the point either...can't you serve the rpc more concurrently with different http addresses more concurrently either (ie maybe a server for each or something)?
< sipa>
separating things off, which can later move elsewhere
< ryanofsky>
doing these things cleanly means planning out url design somewhat, which hasn't been done. in these prs url design is constantly in flux
< sipa>
jtimon: that's completely orthogonal; we're talking about interface here
< jtimon>
well, I remembered that about the most convincing argument in favor of this interface
< jtimon>
s/about/as/
< sipa>
jtimon: we already have multithreaded RPC...
< jtimon>
sipa: oh, nice
< sipa>
jtimon: ...
< ryanofsky>
jtimon, fwiw, i don't see connection there either
< sipa>
jtimon: the RPC implementation isn't very concurrent, but that's an implementation detail that can be improved independently
< jtimon>
I didn't know, sorry, then I don't understand why this wallet/<my wallet> is better than wallet=<mywallet> in the rpc either. but I'll read your conversation and see if I get it
< sipa>
(overeager locking in many places)
< sipa>
jtimon: it's so that things can move to a different process
< sipa>
if the wallet weren't handled by bitcoind anymore, it would have its own RPC server, which runs on a different port or something
< jtimon>
sipa: how is moving things to a different process unrelated to concurrency?
< sipa>
jtimon: because it's already multithreaded!
< jtimon>
then what's the point of moving it to a different process?
< sipa>
the reason for moving things to a different process is for security (don't have your private keys connected to a network interface) and modularity (run it on a different machine)
< jtimon>
ok, thanks, I wrongly assumed the reason was concurrency
< ryanofsky>
my the way my conditional vote for 10849 had an OR in it. if query parameter is worse for some reason (aesthetics?) and path interpretation is way to go, please just document/respond what actual plans are for path interpretation. what compatibility is being promised
< sipa>
ryanofsky: of course it needs documenting
< sipa>
ryanofsky: if we aren't clear on how the namespace separation should happen, i agree that just an RPC named parameter for selecting a wallet is better for now
< sipa>
ryanofsky: but if we are, we should do it... and url parameters seem like a very odd thing for a namespace
< ryanofsky>
i don't understand the point about a namespace. is an aesthetic thing, or a practical difference?
< ryanofsky>
i'm asking for documentation not just because documentation is needed, but also because i really think i am missing something about rationales (or that rationales haven't actually been thought through)
< sipa>
ryanofsky: my concern is that uri parameters don't force us to think about separation
< jtimon>
but the parameters don't need to go in the url, do they? they can go with the json data (ie with method, params, jsonrpc and id)
< sipa>
jtimon: yes, that's the discussion... there are 4 approaches suggested
< jtimon>
I think I'm missing one from the last meeting...
< sipa>
jtimon: in the URL (http://localhost:8333/v1/wallet/[walletname]), in a URL parameter (http://localhost:8333/v1?wallet=[walletname]), in a JSON RPC parameter (pass wallet: "[walletname]" as named argument), through RPC auth (every rpc user has his own wallet)
< jtimon>
I see, rpc auth was the one I was mising
< sipa>
ryanofsky: people may be inclined to add things to URI parameters that don't naturally correspond to separation (say, an option for changing the default currency unit)
< sipa>
ryanofsky: or expect those to go there
< jtimon>
no, I thought we agreed some users will have more than one, and I think some users may not want one too
< sipa>
ryanofsky: sure you can say, "well don't do that"
< ryanofsky>
not sure why a query option like that would be bad at all. do you also think a timeout query option would be bad?
< BlueMatt>
it may be unfair, but ryanofsky succcessfully convinced me over lunch....the user experience of upgrading is gonna be the same no matter how we do it, and the args approach is much simpler for users to add
< BlueMatt>
rather than the endpoints approach
< sipa>
ryanofsky: ok, say we have ?wallet=[walletname]&timeout=30
< sipa>
all in one process
< BlueMatt>
the endpoints stuff is kinda nice in that it makes users thing, but realistically its just another hop to upgrade, cause if we move to process isolation now its all a different ip/port connection anyway
< BlueMatt>
so its just a useless upgrade hop
< sipa>
now the wallets move to different processes
< BlueMatt>
and now the split is by connection port, and not endpoint anyway :p
< sipa>
you can't just go reconfigure the client software
< sipa>
it needs to mix something from the configured URI (which likely has the ?wallet= part) and from the application (which likely has the ?timeout= part)
< BlueMatt>
actually args makes it easier at that point than endpoints?
< BlueMatt>
precisely cause it doesnt require client reconfiguration
< BlueMatt>
wait, now maybe I'm confused, why are we re-adding uri parameters now
< sipa>
but client reconfiguration is easy
< ryanofsky>
sipa are you talking about a node process that forwards requests to various wallets?
< sipa>
ryanofsky: no
< ryanofsky>
BlueMatt, i have same confusion
< ryanofsky>
so you are talking about what matt is talking about. wallet process listening on own port
< BlueMatt>
doing wallet split today is very doable - have multiple rpc clients pointed at different listening ports
< ryanofsky>
wait sipa, that sounds like you are saying one port for node and wallet?
< sipa>
ryanofsky: ???
< sipa>
as long as they're in the same process
< ryanofsky>
port 8333 is wallet process or node process?
< sipa>
it's the process process
< sipa>
there is just one
< BlueMatt>
instead of btc = AuthServiceProxy(); btc.getwalletinfo(); btc.getmininginfo(); its now wallet_btc = AuthServiceProxy(...:8332); node_btc = AuthServiceProxy(...:8331)......
< sipa>
ryanofsky: i'm talking about the case before things move to a idfferent process
< BlueMatt>
sipa: I'm horribly confused
< ryanofsky>
AH ok
< ryanofsky>
so single process just like today
< BlueMatt>
sipa: I think ryanofsky's point (and my point) is that we should focus on making it easier for clients to do the upgrade (which wallet arg is, imo) cause the isolation is gonna come from different listening ports in the future anyway
< sipa>
however, afterwards, when things move to a new process, you don't need to change the application software, just reconfigure it to use a different URI (which is now maybe running on a different port, or a different host)
< BlueMatt>
so trying to force endpoints is just an extra step for users for no reason
< BlueMatt>
sipa: you can already do that?
< ryanofsky>
sipa, how is wallet named arg a problem in that scenario?
< BlueMatt>
just tell your app software that you have two different rpc hosts
< ryanofsky>
i think wallet named arg is a feature even in that scenario
< sipa>
ryanofsky: because wallet named arg requires at the very least the client application to know which wallet it is talking to
< ryanofsky>
because it's easy to imagine starting wallet processes and confusing yourself about port numbers
< sipa>
while it can be encapsulated in the uri
< BlueMatt>
sipa: if you're gonna change the rpc library to support endpoints, you can also change it to support a global wallet argument
< ryanofsky>
i'm confused again. now we are talking about separate wallet process listening it's separate wallet port. what is the harm of wallet param there (used or unused)?
< BlueMatt>
whereas arguments are nice cause you dont have to change the library, if it already supports named args
< sipa>
BlueMatt: but after process separation there may not even be a need for a global wallet argument
< jtimon>
for what is worth, I like v1 in the uri more than "jsonrpc": "1.0" in the json data, but do we need 2 of them?
< sipa>
BlueMatt: endpoint support means there is just one change
< BlueMatt>
sipa: sure, and its, in fact, easier to remove the wallet arg at that point than the endpoints
< sipa>
to the application
< BlueMatt>
cause we can even start ignoring the arg
< sipa>
you really think so? that first requiring everything to add a wallet configuration option, and then later change it again because now it's done in a different URI
< sipa>
is easier than just allowing to configure a URI?
< ryanofsky>
oh sipa, now i finally get it, yes i agree
< BlueMatt>
sipa: think about eg the python library which is never updated
< BlueMatt>
you could use multiwallet even before the library is updated
< BlueMatt>
and if you update the library then its the same
< ryanofsky>
yes that is an argument against named json-rpc params in favor of urls
< BlueMatt>
(either the library silently adds the wallet arg to all calls, or it doesnt)
< sipa>
i'm confused
< BlueMatt>
instead of silently adding the wallet endpoint to all calls, or not
< ryanofsky>
it's not an argument for wallet in uri-path instead of wallet in url query-string
< sipa>
BlueMatt: it shouldn't silently add wallet endpoints!
< sipa>
BlueMatt: you should tell it "MY URL IS ...../v1/wallet"
< BlueMatt>
sipa: does the current python client support that?
< sipa>
it shouldn't even know there is such a thing as wallet names
< BlueMatt>
(I dont know, I'm asking)
< sipa>
BlueMatt: i don't know, but it'd be trivial to add
< sipa>
and certainly easier than updating it to selectively go add wallet parameters everywhere
< sipa>
and then later making that optional
< jtimon>
sipa: how is "encapsulated in the uri" different from "encapsulated on the json data like 'jsonrpc' and 'method'"? if that's better, why not move the method to the uri too?
< sipa>
jtimon: because thr rpc method and parameter are decided by the application
< sipa>
jtimon: and the URI is decided by the person configuring the software
< jtimon>
sipa: and so it's the uri?
< sipa>
no
< sipa>
you tell your application which host and port the RPC server runs on,
< sipa>
you can also tell it what URI to use
< sipa>
ryanofsky: i agree that technically url query-string can do as much as uri-path... but i do believe that this approach can only work if we clearly think about what can be separated
< BlueMatt>
sipa: issue is I do not believe most client libraries support that today
< jtimon>
mhmm, well, that's were I'm confused, I don't see how you could have control of the json data bur not of the uri
< sipa>
BlueMatt: sure, so?
< BlueMatt>
(I was informed that the python-bitcoinrpc library does not)
< sipa>
BlueMatt: they can be updated
< jtimon>
s/were/where
< BlueMatt>
sipa: whereas client libraies may already support a wallet argument
< BlueMatt>
silently, easily
< ryanofsky>
sipa, maybe an example of where query-string would not work & uri-path would work?
< sipa>
18:10:59 < sipa> i guess it isn't worse - as long as everything you'd put in the path becomes a query parameter and nothing else
< ryanofsky>
by the way, one reason i am more inclined to named wallet param is that i think it is actually better for safety
< sipa>
ryanofsky: there isn't
< jtimon>
sipa: oh, ok, then that's where I was confused! what are we discussing then?
< sipa>
jtimon: JSON RPC argument vs URI
< sipa>
jtimon: i'm not talking about URI path vs URI query string
< ryanofsky>
named arguments are safer than positional arguments, because harder to screw up ordering, explicitly specifying wallet is better than not because easy to mess up port numbers
< ryanofsky>
so i tend to think even with multiprocess wallet, param has utility
< jtimon>
sipa: right, so there's no difference between JSON RPC argument vs URI, right?
< sipa>
jtimon: yes there is
< sipa>
jtimon: with URI based approaches, we can avoid the need for applications to know what wallet they're talking to
< sipa>
or even know that there is such a thing as multiwallet
< jtimon>
I thought you just said there isn't an example of what ryanofsky was asking for?
< jtimon>
I'm getting more confused, sorry
< sipa>
jtimon: sigh... that was about URI path vs URI query
< jtimon>
I'll read the full conversation
< ryanofsky>
jtimon, talking about 3 different things. (1) wallet in jsonrpc param (2) wallet in uri query param (3) wallet in uri-path
< jtimon>
I don't think ryanofsky was asking anything about URI query, I know I wasn't
< BlueMatt>
sipa: but thats my point...we cant cause none of the client libraries support it, and they're all like barely maintained last I heard
< sipa>
ryanofsky: every JSON-RPC library supports passing in a URI
< sipa>
eh, BlueMatt ^
< sipa>
BlueMatt: the bitcoin specific shims can be trivially patched to pass that through
< sipa>
adding a named wallet arg everywhere is far harder, and requires logic that imho is totally unneeded at that level
< jtimon>
ryanofsky: and who is defending 2 ?
< jtimon>
ryanofsky: I really thought we were only duscussing 1 vs 3 already, sorry
< sipa>
jtimon: the main discussion is (1) vs (3)... though ryanofsky has suggested that he'd prefer (2) over (3)
< ryanofsky>
i'm defending 2. i think there are practical tradeoffs between 1 & 2, but that 3 has no practical advantages over either and has disadvange of introducing undocumented half-baked url scheme
< BlueMatt>
sipa: you could add a shim that passes the wallet arg in everywhere, too, which is already supported by some rpc client libraries, and all of this debate is useless if we have a process split anyway, cause then its about port numbers anyway
< BlueMatt>
at least then we wont have new endpoints to maintain, just an extra arg that we can ignore
< sipa>
BlueMatt: port numbers are also part of the URI
< jtimon>
ok, I missed that, as said I will read the beginning of the discussion...
< BlueMatt>
yes, but are treated differently by some rpc client software :p
< sipa>
?
< BlueMatt>
well apparently at least python-bitcoinrpc does not support endpoints, but does, obviously, already support a different port to connect to
< BlueMatt>
or so I'm told
< sipa>
yes, but that's easy to fix
< sipa>
and doesn't change that's better if client libraries don't need to know what wallet they're connected to
< jtimon>
yeah, don't know python-bitcoinrpc but I can't imagine how it wouldn't be trivial to adapt either way
< ryanofsky>
BlueMatt, python-bitcoinrpc doesn't support multiple endpoints currently, you have to choose one in advance if you want to write a test that uses multiple wallets for example. but jonas pr changes that
< BlueMatt>
ohoh, wait, so it does support endpoints but you just need multiple AuthServiceProxy's for it?
< jtimon>
afk
< BlueMatt>
wait, that might change my view
< BlueMatt>
ugh
< BlueMatt>
i give up
< ryanofsky>
BlueMatt, exactly
< BlueMatt>
oh, well i mean thats kinda more analygous to the port number changes :(
< BlueMatt>
lol, sorry sipa
< sipa>
ryanofsky: i think that you do bring up a good point that there is a risk in creation a half-baked url scheme
< ryanofsky>
sipa, that is why i am fine with 2
< sipa>
ryanofsky: i believe (2) has more risk in going the wrong way than (3)
< sipa>
and i guess part of that argument is aesthetics
< ryanofsky>
i also want to point out that there we are talking about 1 practical tradeoff between 1 & 2/3, there are other practical advantages to 1 like not having to modify bitcoin-cli
< BlueMatt>
can we use the lsb of the next blockhash?
< ryanofsky>
sipa, ok that is what i think i'm not understanding. is there more to the argument than aesthetics...
< sipa>
ryanofsky: i am fine with (1) over (2)/(3) if we believe the separation isn't thought out enough
< sipa>
ryanofsky: it's a slippery slope argument, so you can counter anything i say with "well, we can just decide not to do that"
< sipa>
ryanofsky: but i believe there is a risk that (2) will make it too easy for us to say "oh, let's put the currency unit in the URI"
< ryanofsky>
sipa, not sure i understand an example of something bad....
< ryanofsky>
you were saying some other parameters are bad to put in query strings?
< sipa>
actually, i change my mind - that wouldn't be bad
< ryanofsky>
also just to list other practical advantages i see in (1) over (2/3). Requires no changes to bitcoin-cli. Is properly documented, easy to understand, just a param. Encourages named arguments. Allows checking wallet name for safety even with multiprocess.
< sipa>
so yes, my preference for (3) over (2) is aesthetics (and, _if_ we have a well thought-out separation, i also think (2) does not really have advantages over (3))
< ryanofsky>
agree maybe (2) may not have advantages over (3) if you have a well-thought uri-path schema
< sipa>
and things like a currency type or timeout can still be query string options rather than paths
< ryanofsky>
i definitely have opposite aesthetics though, straight key=value named argument vastly preferable to me to inflexible /positional/path/stuff
< sipa>
well, (purely aesthetics argument), paths feel like "i am now talking to a different subsystem!"; query options feel like "here is some extra stuff that may or may not affect how you're interpreting my request"
< ryanofsky>
sure, i can see that
< sipa>
and given an intention (if we agree to that) to move the wallet out, i think it makes sense to treat it as a different subsystem
< morcos>
at the risk of derailing this convo on the brink of agreement, sipa: achow: do you still want #10579 for 0.15, it doesn't look as far along as 10571
< ryanofsky>
maybe i don't understand what you see in multiprocess world. i see wallet=filename still being something you can specify for safety
< gribble>
https://github.com/bitcoin/bitcoin/issues/10579 | [RPC] Split signrawtransaction into wallet and non-wallet RPC command by achow101 · Pull Request #10579 · bitcoin/bitcoin · GitHub
< ryanofsky>
but you are seeing us do this elaborate /v1/wallet/ /v1/node thing and then the /v1/wallet stuff goes in the trash because it is no longer a "separate subsystem"?
< morcos>
oops achow101 ^
< sipa>
ryanofsky: no, /v1/wallet would remain - the wallet process just would not expose /v1/node anymore
< ryanofsky>
sipa, ok. that is aesthetically ugly to me :)
< sipa>
ryanofsky: please clarify
< sipa>
(as in, i'd genuinely interested in hearing why)
< sipa>
*i'm
< ryanofsky>
wallet filename at that point is no longer "specifying a subsystem". it is just redundant at that point. we have to go on treating it as this magical thing different from other parameters forever
< sipa>
ryanofsky: well i expect one process to remain capable of handling multiple wallets
< sipa>
a lightweight node is far cheaper than a full node, but handling an individual wallet compared to that is still orders of magnitude less
< achow101>
morcos: I would like to have it in 0.15 but it hasn't been getting any review
< sipa>
ryanofsky: actually, no
< sipa>
ryanofsky: the whole point of having it in a URI is that it shouldn't be treated as part of the interface
< sipa>
someone could create a new lightweight implementation that has the same API (unlikely, i know), which only exposes what we have now under /v1/wallet/[walletname], but just exposes it as '/blah'
< jnewbery>
sorry, I was away from my desk. Seems like I missed out on all the fun.
< sipa>
however, if people feel that we haven't thought through the implication of that, and whether we can do that separation cleanly... we should just do (1)
< jnewbery>
At the risk of going over old ground, I prefer (3) for a couple of reasons:
< jnewbery>
1. each wallet is conceptually a separate resource, so it makes sense to me to have different URIs
< jnewbery>
That's true whether or not we go for wallet separation in future
< jnewbery>
*wallet process separation
< ryanofsky>
thanks jnewbery, you are finally writing the documentatino i was asking for :)
< jnewbery>
2. It offers a smoother upgrade path for clients if we do go for wallet process separation and each wallet binds to its own port
< jnewbery>
namely: change the endpoint now to /wallet/<blah>, and then change the endpoint later to <wallet port>
< jnewbery>
I'm *much* less worried than ryanofsky about implementing a uri scheme now that we want to get rid of later. This should all be considered unstable/experimental anyway
< jnewbery>
yes, I'll happily write documentation
< sipa>
ryanofsky: if anything - thanks for discussing this; it's made your point clearer to me, and helped me understand my own preferences better
< ryanofsky>
your second reason is reason sipa & gmaxwell like approaches (2)&(3) over approach (1), in that respect there is no distiniction between (2)/(3), and countervailing tradeoffs like having to modify bitcoin-cli and other things i mentioned above
< ryanofsky>
and yes, it is clear that i am wayyy more concerned with backwards compatibility than other people
< jnewbery>
In general, I'm also concerned about backwards compatibility. But in this specific case, I'm not
< ryanofsky>
i'm probably just an outlier in that respect
< instagibbs>
jnewbery, a brief(!) recap in the PR itself on the design choice being made would be nice for review/historicity sake
< jnewbery>
> countervailing tradeoffs like having to modify bitcoin-cli
< ryanofsky>
yes named arguments require no changes to bitcoin-cli
< sipa>
ok, i just looked over #10849, and it is suffuciently simple that i don't think i care about implementation complexity compared to 10829
< jnewbery>
I think (2) is better in that respect. We need to modify bitcoin-cli, but we have those changes already coded. With (1), every script that calls wallet methods using bitcoin-cli needs to be changed because positional arguments are no longer supported
< jnewbery>
or have I misunderstood?
< sipa>
jnewbery: that is also a good argument
< ryanofsky>
jnewbery, you understood. i see that is an argument the other way because named arguments are safer
< jnewbery>
with (2), the bitcoin-cli caller needs to change to include a `-usewallet` argument. With (1) the entire invocation needs to change
< ryanofsky>
jnewbery, correct
< jnewbery>
so with (2), it's just as safe. There's no default wallet - it has to be explicitly specified
< ryanofsky>
i'm talking about general safety of named parameters vs positional arguments
< ryanofsky>
i don't think discouraging positional arguments is bad, i actually think it is good
< jnewbery>
oh, absolutely agree there - everyone should use named. But that's an orthogonal point
< sipa>
ryanofsky: that seems orthogonal
< ryanofsky>
i think changes to bitcoin-cli are simple but ugly
< ryanofsky>
anyway these are minor points
< ryanofsky>
i think we all understand tradeoffs between 1 / 2&3 at this point?
< ryanofsky>
for those who have clear notions of what "conceptually a separate resource" means & who don't care about backwards compatibility, there is no difference between 2&3
< sipa>
for those who don't have clear notions, you mean?
< ryanofsky>
i'm saying i don't know clearly what "separate resource" means. it just seems like an arbitrary distinction
< ryanofsky>
it just seems weird to me that you'd want to structure a path scheme around wallet filename, but if you're confident that this is the way to go, then great
< sipa>
how about this: all parts of a request that are expected to be identical between all calls made by a single client application should go into a path
< jnewbery>
Yes, I think we all understand the tradeoffs. My preference order is still 3 > 2 > 1, but I'm happy with any of them going in now, and then coming up with a stable design for all of this in time for v0.16.
< ryanofsky>
sipa, to me it's ugly you even want to make that distinction. aesthetically i prefer if all parts of request should just be treated as similarly as possible
< sipa>
ryanofsky: actually, that sentence applies to (2) and (3) equally
< sipa>
so replace 'path' with 'uri' in it
< ryanofsky>
there are practical reasons (imo weak ones for wanting to make wallet special enough to go in url rather than json request). but making it root of brand new uri schema seems overboard to me
< jtimon>
sipa: I guess we don't move "jsonrpc": "1.0" to the path because it's part of the rpc scheme specification or something?
< sipa>
jtimon: no, it's part of the JSON-RPC spec
< sipa>
oh, that's what you mean; yes
< ryanofsky>
in (1) wallet is just one of many normal params. in (2) wallet is special enough to be a url param. in (3) wallet is root of a new uri-scheme
< jtimon>
right, that's what I meant, but didn't rename json-rpc name
< ryanofsky>
uri-path scheme i mean
< jtimon>
wouldn't putting the wallet there violate the json-rpc spec ?
< sipa>
no
< sipa>
it's just a different URL you're using
< jtimon>
no, I mean, putting wallet in the json data alongside method, jsonrpc, params and id
< ryanofsky>
jtimon, that isn't (1) (2) or (3).
< sipa>
that's not what's being suggested
< ryanofsky>
(1) is just sticking it into params
< sipa>
(1) suggests making the wallet part of 'params'
< jtimon>
oh, I see
< promag>
BlueMatt: saw some rpc functions, all have spaces in ()
< BlueMatt>
promag: hmm, ok
< morcos>
wumpus: not sure where we are with feature freeze, but i think #10707 is going to make it, looks ready for merge
< gribble>
https://github.com/bitcoin/bitcoin/issues/10672 | Avoid division by zero in the case of a corrupt estimates file by practicalswift · Pull Request #10672 · bitcoin/bitcoin · GitHub
< BlueMatt>
we should probably pull an 0.14 and just say that we're frozen with an exception for things already tagged that make it in the next day or two?
< jnewbery>
probably under a different PR number - sorry!
< jnewbery>
I think the prevailing view is we don't need the node sync pause stuff for 0.15
< sipa>
jnewbery: you'd need to do something, though
< sipa>
what is your suggestion?
< jnewbery>
I'm trying to understand what's required. You've mentioned before that HD split makes things worse, but I can't understand why that would be true
< sipa>
it doesn't; i was confused
< jtimon>
so, regarding #9806 txoutsbyaddress, txoutsbyscript and all that...what are the plans for 0.15 again (if any), it seems I may have misunderstood things there
< jnewbery>
so, I'll keep the stuff that marks all keys up to a used key as used
< jnewbery>
gmaxwell says "we could couple that with something that prolongs keeping an encrypted wallet unlocked while syncing/rescanning is running". I'm just looking at how locking/unlocking works to understand what's required there
< sipa>
i think you can just shutdown when the keypool goes below some threshold
< sipa>
which can only happen when you're recovering from an old wallet backup anyway
< jnewbery>
hows that? What if your wallet is encrypted and you can't top up?
< sipa>
then you have a problem
< sipa>
because you're going to silently miss transactions
< jnewbery>
right. Sorry - I don't understand how your keypool can only go below a threshold if you're recovering from an old wallet backup
< sipa>
during normal operation, you never see a key used on the network that you didn't create with getnewaddress, which will make you top up
< jnewbery>
ok, so I get a bunch of adddresses with getnewaddress, I hand them out, my wallet locks, and then I start seeing transactions to those addresses. What next? Isn't my unused keypool running down as I see those transactions?
< sipa>
jnewbery: getnewaddress already marked those keys as used
< sipa>
it can refuse to give you a new one before topping up
< jnewbery>
ah, ok
< sipa>
it does now, but the threshold is 0
< sipa>
that's probably too low, but easy to change
< sipa>
if you're able to avoid hitting a keypool of 0 when its maximum is 100, you're certainly able to avoid hitting 100 when the default is 1000
< BlueMatt>
can we confirm if we need it for 15 or not?
< sipa>
BlueMatt: willdo
< jnewbery>
sipa: if we shutdown when the keypool goes below a threshold, how would I start with an old encrypted wallet that has fewer than the minimum threshold keypool? It'll shutdown before I have a change to unlock the wallet
< sipa>
jnewbery: i don't know
< jtimon>
btw, with gettxout I think it should either include confirmed utxos when calling with include_mempool or have include_mempool renamed to mempool_only or something of the sort. At the very very least s/Whether to include the mempool/Only search in the mempool. Default: true/
< jtimon>
thoughts ?
< sipa>
jtimon: include_mempool is the wrong name
< sipa>
it either presents the view of the utxo at the last block
< sipa>
or the one as seen by the mempool
< jtimon>
right, my point is that if we want to maintain the name, we can first consider the mempool, if not, the current utxo; and the caller can check whether "confirmations" == 0 or not
< jtimon>
but I take it as you prefer just renaming, right?
< sipa>
or properly documenting
< sipa>
but the RPC is about viewing the utxo set... not random access to txouts
< sipa>
we have two utxo sets... the one defined by the blockchain, and the one defined by the blockchain+mempool
< jnewbery>
sipa: I can think of a couple of good solutions to the old encrypted wallet at start: 1. have a bitcoin-wallet util that can topup the keypool offline. 2. have a `loadwallet` RPC that can decrypt on load
< jnewbery>
obviously neither of those are in v0.15. It seems a shame to merge something that could make an encrypted wallet file unloadable
< jnewbery>
there would be one way to recover a blocked wallet: invalidateblock at the wallet's best block, load and unlock the wallet, topup keypool, then reconsiderblock. But I don't think we should be telling users to do that!
< jtimon>
sipa: mhmm, but a confirmed utxo won't appear if you call with include_mempool
< sipa>
jtimon: of course not
< sipa>
it is not an unspent output, when looking at the mempool
< sipa>
wait
< sipa>
i'm not sure what you're saying
< sipa>
jtimon: my answer is about a transaction that is unspent in the chain, but spent by a mempool tx
< jtimon>
right, that won't appear if you chose include_mempoo=true
< sipa>
yes, intentionally
< jtimon>
ok
< sipa>
because it is not an unspent output when looking at the mempool
< jtimon>
oh, I see, but if it's both spent in the chain and the mempool it will appear, I was missing that, thanks
< sipa>
unspent, you mean
< jtimon>
perhaps we should test that too
< jtimon>
yeah, unspent sorry
< jtimon>
ok, so I think I'll write a PR correcting the documentation and testing that case
< sipa>
cool
< jtimon>
but before closing #10822, I would like to know more about what the general thoughts about gettxoutbyaddress, gettxoutbyscript and all that
< sipa>
jtimon: i believe other people have different opinions, but imho that does not belong in core
< sipa>
or at least only if we modularize the code enough so that it's a totally separately pluggable thing
< jtimon>
by thought was to have something like -scriptpubkeyindex analogous to txindex or something like that, but I see you don't like having -txindex already
< sipa>
yes
< sipa>
i'm very strongly opposed to any functionality that requires having a fully indexed blockchain
< jtimon>
of course I don't need this to be in core for my purposes, but since people were talking about it I was trying to find synergies
< sipa>
if it's just the utxo set... it's less bad, but i still think it's beyond our scope
< jtimon>
but a -scriptpubkeyindex only for current utxo would be more acceptable?
< jtimon>
ok
< jtimon>
thanks, I will ask on the next meeting about if I haven't closed #10822 by the time if more people give me their opinion
< bitcoin-git>
[bitcoin] achow101 opened pull request #10857: [RPC] Add a deprecation warning to getinfo's output (master...deprecate-getinfo) https://github.com/bitcoin/bitcoin/pull/10857
< bitcoin-git>
[bitcoin] achow101 opened pull request #10858: [RPC] Add "errors" field to getblockchaininfo and unify "errors" field in get*info RPCs (master...getblockchaininfo-errors) https://github.com/bitcoin/bitcoin/pull/10858