< wumpus> MarcoFalke: agree re 13253
< wumpus> getting the 0.16.1 release out should probably be priority now
< wumpus> moneyball: hahahahaha "double unicorn"
< wumpus> moneyball: but good to know they're making progress. A really persistent one was jtimon's PR that I merged yesterday, #10757.
< gribble> https://github.com/bitcoin/bitcoin/issues/10757 | RPC: Introduce getblockstats to plot things by jtimon · Pull Request #10757 · bitcoin/bitcoin · GitHub
< fanquake> wumpus there are still a couple unclean/complicated backports to do.
< fanquake> Wondering if the people that PR'd the changes to master want to handle the backporting?
< fanquake> wumpus Also if you're happy with it, I think #13246 is ready. I've tested, and it's a good cleanup.
< gribble> https://github.com/bitcoin/bitcoin/issues/13246 | doc: Bump to Ubuntu Bionic 18.04 in build-windows.md by ken2812221 · Pull Request #13246 · bitcoin/bitcoin · GitHub
< bitcoin-git> [bitcoin] laanwj closed pull request #13253: [0.16] Further Backports (0.16...0-16-further-backports) https://github.com/bitcoin/bitcoin/pull/13253
< wumpus> fanquake: I think it's preferable if the people that PRed the changes to master do that, yes
< wumpus> at the very least they'd need to review it carefully
< bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7f4db9a7c354...5c41b6008079
< bitcoin-git> bitcoin/master 9d4f942 Chun Kuan Lee: doc: Bump to Ubuntu Bionic 18.04 in build-windows.md...
< bitcoin-git> bitcoin/master 5c41b60 Wladimir J. van der Laan: Merge #13246: doc: Bump to Ubuntu Bionic 18.04 in build-windows.md...
< fanquake> wumpus I've added them to the "Blockers" in the High Prio list
< bitcoin-git> [bitcoin] laanwj closed pull request #13246: doc: Bump to Ubuntu Bionic 18.04 in build-windows.md (master...patch-2) https://github.com/bitcoin/bitcoin/pull/13246
< fanquake> I think we can just about untag that one for 0.15.2
< kallewoof> Is there a good link to the high prio list? I always have a hard time finding it and keep a tab of https://github.com/bitcoin/bitcoin/pulls?q=is%3Aopen+is%3Apr+project%3Abitcoin%2Fbitcoin%2F8 open, but not sure that URL changes at some point.
< kallewoof> "bitcoin/bitcoin/8" doesn't seem particularly permanent.
< wumpus> fanquake: you mean the GUI settings dialog crash? yes, probably, I don't think it's so likely that we'll do a 0.15.x release anyway
< wumpus> kallewoof: pretty much all github URLs are permanent (for some definitions of permanent at least)
< fanquake> wumpus yep, I'll untag it
< wumpus> fanquake: already did
< fanquake> :o
< wumpus> kallewoof: '8' is the project ID which is not supposed to change
< fanquake> kallewoof I think https://github.com/bitcoin/bitcoin/projects/8 is your best bet for now
< kallewoof> OK, thanks
< aj> ah, helps if you don't get lost in scrollback
< promag> wumpus: #13063
< gribble> https://github.com/bitcoin/bitcoin/issues/13063 | Use shared pointer to retain wallet instance by promag · Pull Request #13063 · bitcoin/bitcoin · GitHub
< wumpus> apparently there's a new "checks" thing on github PRs?
< wumpus> promag: thanks
< fanquake> wumpus yes, not quite sure what that's for yet
< promag> also #13160 could have more feedback
< gribble> https://github.com/bitcoin/bitcoin/issues/13160 | wallet: Unlock spent outputs by promag · Pull Request #13160 · bitcoin/bitcoin · GitHub
< fanquake> I wonder if it's for the linter type things we are currently running on Travis
< promag> fanquake: I guess integrations must update first?
< fanquake> promag Do you mean they will run before travis does?
< promag> fanquake: I mean apps (travis for instance) must update first to use checks
< promag> or we must update .travis.yml?
< fanquake> promag ah ok. I'll have a read up https://blog.github.com/2018-05-07-introducing-checks-api/
< promag> but looks like a nice feature
< fanquake> Also Also, if anyone wasn't aware https://docs.travis-ci.com/user/open-source-on-travis-ci-com
< wumpus> seems like a nice feature, if it can list the failed checks in the PR instead of having to dig into the travis log every time
< wumpus> although having to look for 'apps' in a 'marketplace' seems kind of sily
< fanquake> heh
< wumpus> "
< wumpus> Organization owners and users with push access to a repository can create checks and statuses with GitHub's API. For more information, see "Checks" and "Statuses" in the GitHub Developer documentation."
< wumpus> so we could push *our own* statuses, funny. Though agree with promag it would be useful if this was integrated into, say, travis, to not have to run parallel checking infrastructure.
< bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/5c41b6008079...6378eef18f61
< bitcoin-git> bitcoin/master 80b4910 João Barbosa: wallet: Use shared pointer to retain wallet instance
< bitcoin-git> bitcoin/master 6378eef Wladimir J. van der Laan: Merge #13063: Use shared pointer to retain wallet instance...
< bitcoin-git> [bitcoin] laanwj closed pull request #13063: Use shared pointer to retain wallet instance (master...2018-04-wallet-sharedptr) https://github.com/bitcoin/bitcoin/pull/13063
< promag> wumpus: thanks, #13111 on high priority?
< gribble> https://github.com/bitcoin/bitcoin/issues/13111 | [WIP] Add unloadwallet RPC by promag · Pull Request #13111 · bitcoin/bitcoin · GitHub
< wumpus> I think we should try to keep high-priority discussion in the meetings
< wumpus> that makes sure at least everyone has an idea of what is added. I'm okay with adding one inbetween when there is really a hurry, but this even has a [WIP] tag still
< wumpus> also I think high priority == 0.16.1 now
< promag> wumpus: when is 0.17 feature freeze?
< wumpus> #12624
< gribble> https://github.com/bitcoin/bitcoin/issues/12624 | Release schedule for 0.17.0 · Issue #12624 · bitcoin/bitcoin · GitHub
< promag> tks
< fanquake> wumpus #13284 should be able to go in. Pretty trivial fix.
< gribble> https://github.com/bitcoin/bitcoin/issues/13284 | gui: fix visual "overflow" of amount input. by brandonrninefive · Pull Request #13284 · bitcoin/bitcoin · GitHub
< wumpus> fanquake: will look at it, thanks
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/6378eef18f61...a9b6957383a7
< bitcoin-git> bitcoin/master c865ee1 Wladimir J. van der Laan: Fix FreeBSD build by including utilstrencodings.h...
< bitcoin-git> bitcoin/master a9b6957 MarcoFalke: Merge #13314: Fix FreeBSD build by including utilstrencodings.h...
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #13314: Fix FreeBSD build by including utilstrencodings.h (master...2018_05_freebsd_build) https://github.com/bitcoin/bitcoin/pull/13314
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/a9b6957383a7...536120ec39be
< bitcoin-git> bitcoin/master 97c112d Ben Woosley: Declare TorReply parsing functions in torcontrol_tests...
< bitcoin-git> bitcoin/master 536120e MarcoFalke: Merge #13291: test: Don't include torcontrol.cpp into the test file...
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #13291: test: Don't include torcontrol.cpp into the test file (master...tor-reply) https://github.com/bitcoin/bitcoin/pull/13291
< bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/536120ec39be...f8be43413368
< bitcoin-git> bitcoin/master 5f3cbde Brandon Ruggles: Increased max width of amount field to prevent number overflow bug.
< bitcoin-git> bitcoin/master f8be434 Wladimir J. van der Laan: Merge #13284: gui: fix visual "overflow" of amount input....
< bitcoin-git> [bitcoin] laanwj closed pull request #13284: gui: fix visual "overflow" of amount input. (master...ui_amount_overflow_fix) https://github.com/bitcoin/bitcoin/pull/13284
< fanquake> jonasschnelli I probably should have tested on Windows, but I've almost had enough of VMs for the day
< fanquake> If Windows is broken somehow it'll be the odd one out
< jonasschnelli> fanquake: heh!
< wumpus> ideally it'd use the font size, instead of a fixed width
< wumpus> but an improvement is an improvement...
< wumpus> (but there is a qt function to compute font extents, which we use in some places, if this ever comes up as issue again we should use that)
< bitcoin-git> [bitcoin] FeedTheWeb opened pull request #13315: remove ZMQ message limit (master...master) https://github.com/bitcoin/bitcoin/pull/13315
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/f8be43413368...610f4dd719ad
< bitcoin-git> bitcoin/master fa865ef MarcoFalke: qa: Fix wallet_listreceivedby race
< bitcoin-git> bitcoin/master 610f4dd MarcoFalke: Merge #13304: qa: Fix wallet_listreceivedby race...
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #13304: qa: Fix wallet_listreceivedby race (master...Mf1805-qaWalletRace) https://github.com/bitcoin/bitcoin/pull/13304
< jnewbery> moneyball, jimpo: thanks for continuing to chase Github!
< jnewbery> I also got an email from Ben at Github with another workaround in case people are still having issues: https://0bin.net/paste/4maKKIx04UweS5rA#EvAA3aPuNEb8fRr8x+csVRwfjjS5VCmbP5Mn7r3OGvz
< jnewbery> append ?timeline_per_page=20 to the end of any PR URL
< wumpus> github has a way to restrict the number of posts per page? that's good to know
< promag> jnewbery: #13111 begs for your review =)
< gribble> https://github.com/bitcoin/bitcoin/issues/13111 | Add unloadwallet RPC by promag · Pull Request #13111 · bitcoin/bitcoin · GitHub
< jnewbery> promag: Of course I will, in good time. I've just reviewed #13100, and I don't think there's a huge rush to review all the load/unload wallet PRs in parallel
< gribble> https://github.com/bitcoin/bitcoin/issues/13100 | gui: Add menu entry to open wallet by promag · Pull Request #13100 · bitcoin/bitcoin · GitHub
< promag> jnewbery: what should come first? menu entries or unload?
< jnewbery> I don't think it matters. I'm more concerned about not hogging reviewer/maintainer time.
< jnewbery> I got another update from Ben at Github: One more quick update, the performance improvement I mentioned also made it into production moments ago, so hopefully the URL argument workaround should be less-and-less necessary as well.
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #13317: [0.16.1] Remainig backports (0.16...Mf1805-016ForBackport) https://github.com/bitcoin/bitcoin/pull/13317
< promag> jnewbery: I prefer unloading first, then UI actions can be added at the same time for both load and unload
< sipa> jimpo: in the raw data about bip 158, the second but last column is the number of entries in the basic filter
< sipa> is that the number of unique elements, or the total number? (i.e. does it count multiple identical scriptPubKeys once or multiple times?)
< jimpo> unique elements
< jimpo> if you want the raw data for the sub-filters (as being discussed on the mailing list), I have those too
< sipa> it seems the byte size is almost unbelievably close to the theoretical limit
< jimpo> yeah, 21 is the limit
< jimpo> I have a graph somewhere of average bitsize per element vs # of elements in a filter
< jimpo> it drops off very fast. at around 200 elements or so, I think it's already at 22 or 23 (need to double check that)
< sipa> jimpo: the limit is actually higher
< jimpo> oh? how's that?
< MarcoFalke> I cherry-picked the remaining backports here: #13317
< gribble> https://github.com/bitcoin/bitcoin/issues/13317 | [0.16.1] Remainig backports by MarcoFalke · Pull Request #13317 · bitcoin/bitcoin · GitHub
< MarcoFalke> I believe the conflicts were only due to refactoring (variable names changed) and one adjacent documentation change
< sipa> jimpo: let's discuss after meeting
< wumpus> #startmeeting
< lightningbot> Meeting started Thu May 24 19:00:06 2018 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
< promag> hi
< jimpo> hi
< murch1> hello
< jcorgan> lurking
< wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator
< achow101> hi
< jonasschnelli> hi
< wumpus> I guess the topic of priority today is 0.16.1
< jamesob> hi
< jnewbery> hello
< wumpus> #topic 0.16.1
< wumpus> there's a few backports left to do
< wumpus> or does #13317 include all of them?
< cfields> hi
< gribble> https://github.com/bitcoin/bitcoin/issues/13317 | [0.16.1] Remainig backports by MarcoFalke · Pull Request #13317 · bitcoin/bitcoin · GitHub
< MarcoFalke> wumpus: I left out the qt one
< MarcoFalke> I think it changes translations. I might do it in a separate pull
< wumpus> I'll add that one to "high priority for review" at least
< wumpus> MarcoFalke: ok, thanks!
< wumpus> so please all review that PR ^^
< wumpus> especially the non-trivial backports
< MarcoFalke> With regard to the other high priority prs in the project: I think we can remove the one from BlueMatt for now
< jtimon> https://github.com/bitcoin/bitcoin/pull/12172 is a small thing, but it is a bugfix, perhaps it makes sense to backport it?
< MarcoFalke> And add next week when the rebase is done
< sipa> #12172
< gribble> https://github.com/bitcoin/bitcoin/issues/12172 | Bugfix: RPC: savemempool: Dont save until LoadMempool() is finished by jtimon · Pull Request #12172 · bitcoin/bitcoin · GitHub
< wumpus> I think we should focus on 0.16.1 now, we'll get around to the other high priority stuff again next week
< MarcoFalke> jtimon: that lead to a ton of issues in our tests
< MarcoFalke> I'd prefer we didn't backport that one
< wumpus> I don't think it's terribly urgent to backport, maybe for 0.16.2
< jamesob> is #12431 worth a backport?
< gribble> https://github.com/bitcoin/bitcoin/issues/12431 | Only call NotifyBlockTip when chainActive changes by jamesob · Pull Request #12431 · bitcoin/bitcoin · GitHub
< MarcoFalke> jamesob: What bug does it fix?
< jamesob> potentially incorrect behavior in waitforblockheight (and anything else relying on rpc/blockchain.cpp:latestblock)
< jtimon> MarcoFalke: ok, no hurry, good to know someone tried it, now I'm curious about the issues in the tests and I'll play with travis and the backport
< MarcoFalke> jamesob: waitforblockheight is a hidden tests-only rpc
< wumpus> potentially how bad?
< wumpus> oh
< jnewbery> jtimon: #12863
< gribble> https://github.com/bitcoin/bitcoin/issues/12863 | mempool_persist.py failing in travis · Issue #12863 · bitcoin/bitcoin · GitHub
< jtimon> jnewbery: thanks
< wumpus> other topics?
< sipa> i want to bring up #13298 briefly
< gribble> https://github.com/bitcoin/bitcoin/issues/13298 | Net: Random delays *per network group* to obfuscate transaction time by naumenkogs · Pull Request #13298 · bitcoin/bitcoin · GitHub
< wumpus> anything else important for 0.16.1 that still needs to be done?
< sipa> (not for 0.16.1, to be clear)
< wumpus> (no problem, that was an orthogonal question)
< wumpus> #topic Random delays *per network group* to obfuscate transaction time
< sipa> i want to bring it up because it's a possibly significant effect on P2P transaction relay
< sipa> and it needs review beyond "does the code work"
< wumpus> I haven't had chance to look at it yet
< sipa> but it's also just local policy and not something that warrants a BIP imho
< wumpus> agree
< sipa> maybe there's not much more to say about that, just hoping to get people to think about it a bit
< wumpus> #action look at PR 13298
< wumpus> it's three days old, so people are excused not having an opinion on it yet :-)
< sipa> end topic :)
< cfields> thanks for bringing it up, I'll have a look as well...
< wumpus> other topics?
< cfields> it'd be nice to have a tool to model that kind of change, though.
< provoostenator> Topic suggestion: GUI prune setting
< wumpus> cfields: makes sense, maybe propose it in the PR
< wumpus> #topic GUI prune setting (provoostenator)
< provoostenator> AKA #13043
< gribble> https://github.com/bitcoin/bitcoin/issues/13043 | [qt] OptionsDialog: add prune setting by Sjors · Pull Request #13043 · bitcoin/bitcoin · GitHub
< jonasschnelli> Looks good,.. maybe orthogonal, but the prune settings should be in the intro...
< provoostenator> There's some confusion around whether using QT settings is appropriate for this, and I see three ways out.
< jonasschnelli> That's where it is probably most valuable
< provoostenator> 1. ignore the problem
< wumpus> I'm ok with most solutions, except writing to bitcoin.conf
< provoostenator> 2. go the writable config file route
< jonasschnelli> what wumpus sais
< kanzure> hi.
< provoostenator> 3. interpret a lack of prune= setting differently
< jonasschnelli> Can't the GUI settings not just override what is already set?
< instagibbs> (3) is interesting
< achow101> there currently are a few settings that are saved to qt settings that are shared between qt and bitcoind
< achow101> but bitcoind can't access
< provoostenator> If we go for (2) then I'd like to nominate #11082 for priority review
< gribble> https://github.com/bitcoin/bitcoin/issues/11082 | Add new bitcoin_rw.conf file that is used for settings modified by this software itself by luke-jr · Pull Request #11082 · bitcoin/bitcoin · GitHub
< jonasschnelli> If prune is set via confi/startip, disallow access in the GUI settings (only display it)
< achow101> so if we follow what has been done previously, then we ignore it
< wumpus> yes an additional writable config file would be fine
< jonasschnelli> Do we want four(!) levels of configuration?
< jimpo> What about augmenting the GUI to help you generate a default config file when none already exists?
< jonasschnelli> And eventually importing conf-files?
< wumpus> there have also been plans to add RPCs to change configuration settings, that'd require a similar thing, just don't write the -conf file it's just as likely to be in an read-only directory
< jonasschnelli> Would the rw_conf file be replacing the QSettings layer?
< wumpus> jonasschnelli: probably
< provoostenator> Right, I'd also like to get rid of QT settings completely and use a read-write (seperate) config file.
< wumpus> jonasschnelli: long term, at least
< jonasschnelli> That would be acceptable
< provoostenator> I wrote a migration away from QTSettings here: #12833
< gribble> https://github.com/bitcoin/bitcoin/issues/12833 | [qt] move QSettings to bitcoin.conf where possible by Sjors · Pull Request #12833 · bitcoin/bitcoin · GitHub
< jonasschnelli> But please not conf<->startup-cli-params<->QTSettings<->level4_rw_conf
< wumpus> nice
< jonasschnelli> Thanks provoostenator for working on that!
< achow101> since this is a problem that effects multiple options, can we just ignore the problem for now and deal with them all together at the same time with a better solution?
< wumpus> yes storing some of the settings in a different place has been problematic
< wumpus> (at least in the QSettings - because bitcoind can't get there)
< instagibbs> users could by and large be migrated over the rw unless they have a need for read-only
< instagibbs> for simplicity
< wumpus> the only thing that would idally be stored in the QSettings would be the data directory
< wumpus> well the bitcoin.conf is for human editing
< provoostenator> This migration would also work if it's done _after_ we add prune stuff to QTSettings.
< wumpus> the _rw is machine writable, all comments will be discarded etc
< instagibbs> ah hm
< jonasschnelli> yes
< provoostenator> But if we can get the proper solution over with, that might be better.
< jonasschnelli> For 12833 I'm also unsure about the term "Limit".
< wumpus> on the other hand, having things be dependent on each other is usually a bad idea, just draws out things
< jonasschnelli> Since this is not true
< provoostenator> jonasschnelli: what "Limit"?
< jonasschnelli> provoostenator: 12833 currently tells user "Limit block storage to: " which is not true... though the tooptip hints towards the right handling.
< wumpus> how is it not true?
< provoostenator> That's #13043
< gribble> https://github.com/bitcoin/bitcoin/issues/13043 | [qt] OptionsDialog: add prune setting by Sjors · Pull Request #13043 · bitcoin/bitcoin · GitHub
< provoostenator> Which I fixed, but screenshot is outdated.
< provoostenator> It's now "Prune &amp;block storage to"
< jonasschnelli> wumpus: AFAIK the 550MB assumption (if one sets 550) is still on 1MB blocks
< jonasschnelli> But 288min blocks & undos are enforced
< wumpus> jonasschnelli: right, that's an issue for the command line help too though, I think?
< jonasschnelli> but maybe we should fix that (if it turns out to be a problem) rathern then changing the word "limit" :)
< provoostenator> I'm not too worried about details of the text and minimum size. It's what we need to do about saving settings that seems to cause things to get stuck.
< jonasschnelli> Indeed
< wumpus> other topics?
< jonasschnelli> I also hoped we could educate the user withing the GUI a bit more about pruning... but can be done later via extending the intro.cpp
< wumpus> right, let's focus on getting the functionality in first. I think educating the user is a good thing, but having the PR depend on all those things being worked out is going to put it way past 0.17 probably.
< jonasschnelli> I have a short topic: sipa raised concerns about the scantxoutset RPC command
< jonasschnelli> (if that is something we want to discuss here)
< wumpus> so the feature freeze for 0.17 is 2018-07-16, less than two months away
< provoostenator> Also note that I don't think Mac shows the intro dialog.
< wumpus> #topic scantxoutset RPC command (jonasschnelli)
< jonasschnelli> provoostenator: it does at least for me
< wumpus> provoostenator: it should!
< provoostenator> jonasschnelli: ok, maybe I need to delete more stuff for it to appear.
< jonasschnelli> Before continuing on #12196 we may want to discuss it it makes sense to have a scantxoutset command
< gribble> https://github.com/bitcoin/bitcoin/issues/12196 | Add scantxoutset RPC method by jonasschnelli · Pull Request #12196 · bitcoin/bitcoin · GitHub
< wumpus> you need to delete -whereever it stores the qsettings on Mac-
< wumpus> on UNIX you can pass the flag -resetguisettings but that's not easy on Mac I think
< jonasschnelli> (works on mac as well)
< wumpus> good :)
< provoostenator> Resetting gui settings didn't do it for me, but will debug some other time.
< achow101> jonasschnelli: what is the command supposed to do?
< jonasschnelli> The scan functionality allows utxo sweeping (rawsweeptransaction) with no block scanning
< jonasschnelli> You can pass in n pubkeys/addresses or even xpubs with a lookup window and it gives you back all unspents
< sipa> yeah i just mentioned that we preferably don't commit to having functionality that's hard to maintain in the future
< jonasschnelli> And eben a rawsweeptransaction to a single address
< sipa> fun.
< instagibbs> err what
< wumpus> services massacre
< cfields> irc unicorns...
< cfields> let's move to slack!
< cfields> (/s)
< wumpus> :-(
< * jonasschnelli> stabs cfields
< sipa> back to topic?
< sipa> yeah i just mentioned that we preferably don't commit to having functionality that's hard to maintain in the future
< jonasschnelli> Yes. I think sipa's point is a valid point. Do we want to maintain something that may be incompatible with future utxo handling (or new model=
< provoostenator> Such functionality seems quite useful for watch-only stuff
< provoostenator> Without actually having to import things into a wallet.
< sipa> which isn't a problem if it were implemented on top of an optional index
< jimpo> sipa: is the optional index a full scriptPubKey index?
< provoostenator> But without caching of some sort (or an index?) I'm guessing it'd be very slow.
< jimpo> or something less than that?
< jonasschnelli> sipa: what if we allow it for now an mention in the RN it may later require an optional index?
< sipa> yes, perhaps
< jonasschnelli> provoostenator: 30seconds for the whole index with an xpub & 1000 keys lookup window on a SSD/fastCPU machine
< jimpo> Or just have an explicit flag enabling the RPC now, even if it requires no additional index at present?
< jonasschnelli> *whole set
< sipa> yeah, that seems fine
< sipa> i do see the usefulness of scanning the UTXO set
< provoostenator> All the way back to the genesis block?
< jonasschnelli> jimpo: yes. But feels a bit after artificially holding back functions due to possible future work (which may never happen)
< sipa> provoostenator: it scans the UTXO set, not the blockchain
< wumpus> yes, scanning the utxo set would be incredibly useful
< wumpus> I've wanted this functionality since forever really
< provoostenator> Ah OK, it just gets unspends, not transaction history. Well that's still quite useful indeed.
< jonasschnelli> I wanted that in master before the fork-coins happend. :)
< wumpus> even if slow it's so much faster than scanning the entire chain
< wumpus> (without index)
< jonasschnelli> it would also allow importing wallets (no rescan required) if one don't care about the tx history
< wumpus> using importprunedfunds I guess?
< jonasschnelli> Yes..
< jonasschnelli> conclusion? Hide behind an artificial block-setting or risk it will not maintainable over time?
< jimpo> I think it's probably fine to risk it not being maintainable over time without an explicit index
< jonasschnelli> Yes. I would feel okay with that...
< jonasschnelli> Let me mention that risk in the RN and fix the PR in general /topic
< wumpus> so it will just become slower due to the linear scan?
< provoostenator> By "not being maintainable over time" do you mean if the UTXO set gets really large or is it a code maintenance things?
< jonasschnelli> from sipa:
< jonasschnelli> Overall, I'm unsure about this. This is functionality that is more easily provided by software that maintains a UTXO index by script, and is not possible in general if we'd move to a design like UHF (see mailinglist) or other UTXO avoidance techniques. Those are far away of course, and features like this can be made optional (like txindex is) if needed. I'm just generally unconvinced a full node is the best place to put
< jonasschnelli> this.
< wumpus> or is this 'unmaintainable' as in 'will give wumpus more headaches'?
< jonasschnelli> no.. changes of the general model
< sipa> wumpus: in a UHF model, without indexes, implementing a scan of the UTXO set requires going through the blockchain
< sipa> (where we store hashes of UTXOs rather than the UTXOs themselves)
< instagibbs> sipa, searchable index has been tried a few times, and sadly dropped, something like 3 times
< wumpus> sipa: so that's a far future thing right?
< instagibbs> not saying it's not the right way
< sipa> instagibbs: yes, my preference is that's implemented in other software
< jcorgan> i bet there's a dozen private implementations of tx/addr external indexing for xpub related things
< wumpus> I'd really like a way to scan UTXOs, my own appraoch was to stream the UTXO set over HTTP and do it client-side, but that ran into problems with the libevent http server :(
< jonasschnelli> Yes. But there is no fast access to the UTXO set from outside of our code-base IMO
< jcorgan> i do it with zmq notifications and the REST interface
< * jonasschnelli> curses zmq
< jcorgan> you're welcome :-)
< * wumpus> still wants to resurrect #7759 some day
< gribble> https://github.com/bitcoin/bitcoin/issues/7759 | [WIP] rest: Stream entire utxo set by laanwj · Pull Request #7759 · bitcoin/bitcoin · GitHub
< achow101> I've taken to modifying gettxoutsetinfo whenever I need to scan the utxo set. takes like 20 minutes though to scan the whole thing
< jonasschnelli> interesting... almost forgotten
< provoostenator> In this future where "we store hashes of UTXOs rather than the UTXOs themselves", wouldn't there simply be an optional "index" with the UTXO, which this RPC method could then move to?
< jonasschnelli> achow101: only if you are in debug mode? right... takes 30secs here
< wumpus> provoostenator: yes... exactly... I say it's a problem/option for then
< provoostenator> We'd have to make it clear that in the future the method would / might require an index.
< achow101> jonasschnelli: I think it was the operations I was doing, or my slow hdd
< wumpus> achow101: on ARM it's pretty slow (but still faster than scanning the whole chain!)
< sipa> it's the same issue as we had with txindex
< sipa> people build solution that assume txindex is always there... then we moved to a UTXO model
< sipa> and txindex became an inefficient optional thing
< wumpus> we can even deprecate the RPC then
< achow101> sipa: then it will be the same situation as getrawtransaction
< jimpo> with #13243, it should become less costly to build future indexes in the background...
< wumpus> I don't think we should reject useful, optional, functionality just because of some future data structure change
< gribble> https://github.com/bitcoin/bitcoin/issues/13243 | Make reusable base class for auxiliary indices by jimpo · Pull Request #13243 · bitcoin/bitcoin · GitHub
< sipa> achow101: my concern is not the incompatibility
< sipa> achow101: my concern is people building an ecosystem that assumes it's always possible and cheap to do
< sipa> but okay, i agree with the points here
< wumpus> sipa: I agree with that in general, but I'm not sure here
< jonasschnelli> jimpo: great work!
< provoostenator> The fact that it takes 30 seconds is helpful in that case. :-)
< jcorgan> if we want to encourage people to treat bitcoind as the "ground truth", instead of baking up their own stuff, giving them easier access to the "database" would help
< sipa> jcorgan: yes... except that the ground truth in the future may not be the UTXO set
< provoostenator> jcorgan: that too, would be nice to be able to get easy to export dumps of useful things, though not should what format.
< jcorgan> sure, but anything can happen in the future
< wumpus> sure, but anything can happen in the future <- that
< jonasschnelli> jimpo: the question is, if we not want to build a base for an external indexing daemon (outside of the Core project)
< wumpus> in any case this is hard to do out-of-process right now!
< wumpus> even harder than indexing
< jcorgan> i'd be happy with that, too, instead of making everyone else recreate it
< jonasschnelli> wumpus: Indeed
< sipa> it's also less a concern to add optional indexes now with jimpo's background index work
< sipa> before, new indexes always required ugly hacks all over the validation code
< wumpus> nice!
< jimpo> My guess is there will be ongoing tension between adding RPC functionality and keeping the node requirements small unless there are more options for users
< jcorgan> yes
< echeveria> it's a really bad idea to add an address index.
< provoostenator> Not too mention that you couldn't turn on/off txindex without reindexing everything.
< instagibbs> jimpo, an rpc call in every garage!
< jcorgan> i used to maintain the addrindex patch set and it got uglier and uglier over time
< echeveria> it means people will willyfully build insane systems.
< jimpo> haha
< sipa> echeveria: yes :(
< jonasschnelli> echeveria: agree
< sipa> but i fear they'll do so anyway
< echeveria> rather than sane, scalable things that don't require an ever growing index.
< jimpo> I tend to agree a better solution is to have a separate indexing service that doesn't do consensus but maintains the full chain state
< jonasschnelli> The question is, is it faster to index everything or to have Core running with 10k wallets
< jcorgan> +1 jimpo
< jimpo> and gets blocks via zmq
< sipa> decent wallet software shouldn't ever need to scan
< sipa> except for recovery
< jcorgan> i want to let bitcoind do the hard stuff (validation)
< jcorgan> but then i want to easily get at all the validated data
< wumpus> maybe it never *needs* to, but there are legit situations in which it's useful as a tool
< sipa> sure!
< jonasschnelli> jimpo: fork Core, ripout the validation stuff and you have an indexing daemon you can connect to your core node over p2p
< sipa> having a way to disable validation would also help with that :)
< sipa> anyway, this is turning into a philosophical discussion
< wumpus> jonasschnelli: why not use your indexing daemon?
< wumpus> yes, I think we're through the meeting
< jonasschnelli> wumpus: maybe. Not sure if we want another p2p library introduced
< wumpus> jonasschnelli: not into bitcoin core, I mean as external thing
< wumpus> #endmeeting
< lightningbot> Meeting ended Thu May 24 19:59:27 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< jonasschnelli> wumpus: yes. But even there we may want to reuse the primitives...
< jonasschnelli> or maybe its good to use another library to find things we would not find using the same core code
< jimpo> yeah, I think diversity at the P2P layer is healthy, using the same validation engine
< jimpo> and some can have features requiring additional indexes
< jimpo> speaking of which, what were you saying about the BIP 158 compression ratios, sipa?
< sipa> jimpo: sec
< wumpus> jimpo: rust P2P layer? *ducks*
< jimpo> of course
< jimpo> :-)
< sipa> jimpo: so say you have a set of N elements
< sipa> (after deduplication)
< sipa> this means you have a list of N entries, each in range 0..2^20*N-1, whose order does not matter
< sipa> the number of combinations for that is (2^20*N)^N / N!
< sipa> (approximately; this formula ignores that there may be duplicates among those N, but that chance is low)
< sipa> this means that information theoretically, you need at least log2((2^20*N)^N / N!) bits in total
< sipa> otherwise you couldn't express every combination
< sipa> for N=10000, that number is 214419 bits, or 21.4419 bits per element
< provoostenator> When I do "bitcoin-cli -config=/the/usual/place -datadir=/some/other/place getblockheight" it creates a folder /some/other/place/wallets
< sipa> jimpo: a GCS implementation will use 21.5819 bits on average
< jimpo> oh, interesting
< provoostenator> Even if the RPC connection fails.
< jimpo> (damn it small graphs on Wolfram Alpha)
< sipa> jimpo: what i don't know is if there's isn't another probabilistic data structure that has a 1/2^P false postive rate which needs less information
< sipa> but at least any construction that follows from compressing a list of N elements in range 0...2^20*N will at least need 21.4419
< jimpo> thanks for the explanation
< sipa> but this is *really* good
< jimpo> yeah, seems GCS is doing very well
< sipa> less than 1% overhead above the theoretical minimum
< jimpo> I've been thinking a bit about tuning the P value. Kind of unfortunate that it's static.
< sipa> even at just 100 elements, GCS has less than 1% overhead
< sipa> at a false positive rate of 1/2^10 it's close to 1.5% overhead
< sipa> gmaxwell and i were looking at whether a better custom entropy coder could do better than GCS, but then he pointed out to me that the limit isn't 20 bits per element, and that your numbers looked suspiciously close to the limit already
< jimpo> what is a custom entropy coder?
< sipa> not using golomb-rice coding but something custom that could get us closer to the theoretical limit
< sipa> it's certainly possible with a range coder
< sipa> though that would be complex and computationally expensive
< sipa> and now that it looks like golomb coding is so close to optimal already, it doesn't look worth looking into
< jimpo> out of curiosity, have you looked at PFOR/FastPFOR for integer sequence compression?
< sipa> i have never heard about those
< jimpo> I was experimenting with it yesterday for compression header values
< jimpo> and it gets pretty amazing results
< jimpo> if you take all off the versions in a 3,000 header range in a sequence, and do the same for timestamps, bits, and nonces, it compresses all of them together by ~90%
< jimpo> can even compress sequences of nonces 75% on average
< echeveria> that seems kind of unlikely.
< jimpo> yeah, that's what I thought. but I tried decompressing some of the ranges and it seems to work?
< bitcoin-git> [bitcoin] TronBlack opened pull request #13318: Remove legacy verification (master...Remove-legacy-verification) https://github.com/bitcoin/bitcoin/pull/13318
< echeveria> lol
< echeveria> what on earth happened there. 30 second old pull request with 2 month old comments.
< harding> echeveria: the comments are on the specific commits, not the PR itself.
< echeveria> ah, github presents that confusingly.
< bitcoin-git> [bitcoin] TronBlack closed pull request #13318: Remove legacy verification (master...Remove-legacy-verification) https://github.com/bitcoin/bitcoin/pull/13318
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #13319: gui: Backport bech32 checkbox (0.16...Mf1805-016ForBackportGui) https://github.com/bitcoin/bitcoin/pull/13319
< phantomcircuit> i've got a reeeeally old wallet
< phantomcircuit> the result of getbalance doesn't match the sum of all amounts in listtransactions "*" 1000000
< phantomcircuit> seems like a bug
< sipa> any unconfirmed transactions/
< phantomcircuit> sipa, nope
< phantomcircuit> possibly it's cause it's all the dust that someone was flooding a while ago?
< sipa> you're calling getbalance, not getbalance "", right/
< sipa> jimpo: it seems PFOR assumes integers are sorted?
< sipa> oh, or just the benchmark
< phantomcircuit> sipa, im running git master so the accounts stuff has been removed
< phantomcircuit> and im using the multi wallet support
< phantomcircuit> i guess i need to test this on stable
< phantomcircuit> sipa, hmm listunspent matches getbalance of course
< phantomcircuit> sipa, that wasn't it
< phantomcircuit> the sum of the amounts field in listtransactions nominally should match getbalance i believe
< gmaxwell> coinjoins fuxor that no?
< phantomcircuit> gmaxwell, shouldn't i dont think
< gmaxwell> when you 'sum amounts' how are you handling fees?
< phantomcircuit> hmm maybe it's that let me check
< phantomcircuit> derp yeah that was it
< gmaxwell> Next customer!
< sipa> queue = rotl(queue, 1)
< * echeveria> sets mode +b phantomcircuit