< kallewoof> RFC: I started on a file format for storing mempool history -> https://bc-2.jp/mempool%20file%20format.txt (sorry if off-topic)
< fanquake> kallewoof sounds on-topic
< kallewoof> fanquake: Thanks, it felt borderline. Btw did that gist work for you?
< fanquake> kallewoof Sorry that's the next thing I'm working on. Got stuck inside Windows reviewing :(
< kallewoof> fanquake: NP!
< sipa> complete.
< aj> kallewoof: gist worked for me fwiw (i'd missed running autogen under scan-build)
< kallewoof> aj: Great!
<@wumpus> kallewoof: IMO it' technical and specific enough to be on-topic
< kallewoof> wumpus: Okay. Technical+specific makes sense to me.
<@wumpus> you're not the first to do this but probably the first to try to formalize the data format. There's also some synergies with e.g. the zmq notification protocol here.
< aj> kallewoof: fwiw, i have about 12GB of compressed mempool tx data from the last few months, with the idea to some day analyse if any of the non-confirming txes display any interesting behaviour
< aj> kallewoof: the 1-satoshi-is-plenty phase started shortly after i started keeping track though, so it's probably boring :(
<@wumpus> after all, a log format is pretty much *concatanate all the notifications*
< aj> kallewoof: (my data is basically hexified zmq data, with a timestamp, then lz compressed)
< aj> kallewoof: includeconf travis -- test_runner is failing travis of the warning stderr output i think?
< kallewoof> aj: Oh, you have data? I would love to get my hands on it!
< kallewoof> wumpus: I figured, yeah. Having a formalized format would be beneficial I think, unless the format is crap. (Hence RFC :) )
<@wumpus> kallewoof: yes
< kallewoof> aj: Ah... of course. I'll switch to regular printf()..
< kallewoof> wumpus: I should def look at zmq notification protocol, yeah. I forgot about that.
<@wumpus> kallewoof: but you have a good point I think the difference conceptually is that the zmq notification protocol is specific to bitcoin core, and includes some specific implementation details, while you're trying to make a more general exchange format
<@wumpus> (e.g. we have many more mempool retirement reasons, but some of those are very specific)
< fanquake> kallewoof Followed static analyzer your steps, seems to be working correctly. Cheers.
< kallewoof> wumpus: What other than 'old not confirmed', 'rbf-replaced', and 'double-spent-in-block'?
< kallewoof> fanquake: Cool :)
<@wumpus> kallewoof: see MemPoolRemovalReason enum
< kallewoof> wumpus: Will do!
< aj> kallewoof: http://azure.erisian.com.au/~aj/tmp/kalle/ has data from yesterday (24 hours, 1000 UTC - 0959 UTC ish) and the logging script
< aj> kallewoof: most of the data's behind my home internet which will make getting it out usefully less than easy :)
< kallewoof> aj: Nice. Downloading
< kallewoof> aj: Would access to an ssh machine somewhere be helpful?
< kallewoof> s/ssh machine/ssh account/
< aj> kallewoof: maybe? somewhere with disk to put it would be helpful; otherwise it's just upstream-b/w-bound
< kallewoof> aj: Wasn't sure if your problem is bandwidth or simply don't have a disk online with enough space. I can give you access to a digital ocean machine somewhere.
< kallewoof> aj: I got the files. Thanks, will check soon :)
< aj> kallewoof: well, probably be more useful to see if the data's actually interesting; b/w and storage i can solve myself without too much hassle :)
< kallewoof> aj: Makes sense yeah
< jonasschnelli> If someone is interested, one of my machines does dump the mempool (mammempool 300MB) every hour since a couple of month... a lot of data though
<@wumpus> the incremental format is probably slightly more efficient in storage, sounds like dumping the entire thing every hour will get you a lot of duplicate data, though compression helps ofc
< jonasschnelli> Yes! I had to trade off storage space vs. implementing an incremental dump solution
<@wumpus> yep
< fanquake> jonasschnelli Thanks
< jonasschnelli> fanquake: I need to finish my IRC bot that allows triggering a gitian build on my machine
< jonasschnelli> Building all pull requests would require a couple of machines (=much more expensive setup)
<@wumpus> indeed, I think it's better to have it as something manually triggered, at least at first, can always be automatized if there's need for it
< jonasschnelli> Also, there are risks running pre-built binaries...
<@wumpus> yes, there are security issues with building ,as well as hosting binaries for PRs that no human has looked at
< fanquake> handy way for someone to submit a malicious PR and get their binaries built ready for distribution
<@wumpus> exactly
<@wumpus> it does leave one hell of an audit trail, so I doubt it's the best way to parasitize on someone else's infrastructure to host binaries, but still we really want to avoid it
< jonasschnelli> Trigger a build could be protected over a github username (via special comment) or via an IRC handle. Both not very secure... additionally a gpg signature coule be added
<@wumpus> jonasschnelli: well as long as you check that freenode user is authenticated and using TLS, your only worry is that freenode itself is compromised, it's unlikely anyone will go that far to trigger a build :-)
< jonasschnelli> heh, true.
<@wumpus> same for spoofing github handles I think
< fanquake> Probably easier to spin up some EC2 instances using credit cards before compromising the freenode infrastructure
< fanquake> *stolen credit cards
< fanquake> wumpus Did you want to merge #12715 tomorrow or the day after? I've just about finished testing, looks like it's ready to go. Just want to minimise disruption.
< gribble> https://github.com/bitcoin/bitcoin/issues/12715 | depends: Add make clean rule by hkjn · Pull Request #12715 · bitcoin/bitcoin · GitHub
< bitcoin-git> [bitcoin] jonasschnelli opened pull request #13014: Allow txindex in prune mode (master...2018/04/txindex_prune) https://github.com/bitcoin/bitcoin/pull/13014
< jonasschnelli> wow... I can load #11857 again
< gribble> https://github.com/bitcoin/bitcoin/issues/11857 | Build tx index in parallel with validation by jimpo · Pull Request #11857 · bitcoin/bitcoin · GitHub
< jonasschnelli> no "github unicorn"
< luke-jr> (I can't)
< fanquake> (neither)
< jonasschnelli> hmm...
< jonasschnelli> I don't see the reason why this PR needs more time to load...
< jonasschnelli> Maybe we reopen a new PR (could be a comment that blocks it) or jimpo tries to rebase and force push
< jonasschnelli> It's a bit odd that we have a high-prio PR that can't be discussed.
< fanquake> It's loading now, so it must just be some temporary issue.
< jonasschnelli> fanquake@testing is on fire!
< bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/3a8a4dc4a130...8fd62437c683
< bitcoin-git> bitcoin/master aff16fd Henrik Jonsson: depends: Add 'make clean' and 'make clean-all' rules...
< bitcoin-git> bitcoin/master 8fd6243 Wladimir J. van der Laan: Merge #12715: depends: Add 'make clean' rule...
< bitcoin-git> [bitcoin] laanwj closed pull request #12715: depends: Add 'make clean' rule (master...clean-depends) https://github.com/bitcoin/bitcoin/pull/12715
< bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/8fd62437c683...0d12570a8037
< bitcoin-git> bitcoin/master d41a420 João Barbosa: test: Fix dangling wallet pointer in vpwallets
< bitcoin-git> bitcoin/master 0d12570 Wladimir J. van der Laan: Merge #13007: test: Fix dangling wallet pointer in vpwallets...
< bitcoin-git> [bitcoin] laanwj closed pull request #13007: test: Fix dangling wallet pointer in vpwallets (master...2018-04-fixwallettest) https://github.com/bitcoin/bitcoin/pull/13007
< fanquake> wumpus In agreement to close #12990 ?
< gribble> https://github.com/bitcoin/bitcoin/issues/12990 | depends: Fix libX11 build on gcc 8 by MarcoFalke · Pull Request #12990 · bitcoin/bitcoin · GitHub
<@wumpus> <jonasschnelli> fanquake@testing is on fire! <- yeah, thanks for all the testing!
<@wumpus> fanquake: will take a look
<@wumpus> fanquake: I was leaving that to marcofalke, but I guess, as we all seem to agree we can just close it, someone who needs the patch right now could always pull that branch I guess...
< bitcoin-git> [bitcoin] laanwj closed pull request #12990: depends: Fix libX11 build on gcc 8 (master...Mf1804-dependsGCC8Fix) https://github.com/bitcoin/bitcoin/pull/12990
< bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/0d12570a8037...615f7c288414
< bitcoin-git> bitcoin/master 7d8a8cc JeremyRand: Avoid launching as admin when NSIS installer ends....
< bitcoin-git> bitcoin/master 615f7c2 Wladimir J. van der Laan: Merge #12985: Windows: Avoid launching as admin when NSIS installer ends....
< bitcoin-git> [bitcoin] laanwj closed pull request #12985: Windows: Avoid launching as admin when NSIS installer ends. (master...nsis-de-elevate) https://github.com/bitcoin/bitcoin/pull/12985
< michagogo_> wumpus: #12985 may be problematic
< gribble> https://github.com/bitcoin/bitcoin/issues/12985 | Windows: Avoid launching as admin when NSIS installer ends. by JeremyRand · Pull Request #12985 · bitcoin/bitcoin · GitHub
< bitcoin-git> [bitcoin] Empact closed pull request #12991: Remove unused default args to Invalid and DoS (master...remove-unused-default-args-dos) https://github.com/bitcoin/bitcoin/pull/12991
< bitcoin-git> [bitcoin] Empact opened pull request #13016: scripted-diff: Rename CChainState::g_failed_blocks to m_failed_blocks (master...g-failed-blocks) https://github.com/bitcoin/bitcoin/pull/13016
< bitcoin-git> [bitcoin] promag opened pull request #13017: Add AddWallet, RemoveWallet, GetWallet and GetWallets (master...2018-04-vpwallets) https://github.com/bitcoin/bitcoin/pull/13017
< bitcoin-git> [bitcoin] Empact opened pull request #13018: Make AbortNode function static (master...abort-node-static) https://github.com/bitcoin/bitcoin/pull/13018
< bitcoin-git> [bitcoin] Empact closed pull request #13018: Make AbortNode function static (master...abort-node-static) https://github.com/bitcoin/bitcoin/pull/13018
< Nurlan23> Hi ya all!
< jamesob> jonasschnelli luke-jr fanquake: #11857 loads pretty reliably when emulating a mobile device using chrome's developer tools :)
< gribble> https://github.com/bitcoin/bitcoin/issues/11857 | Build tx index in parallel with validation by jimpo · Pull Request #11857 · bitcoin/bitcoin · GitHub
< instagibbs> huh, loaded it in incognito browser, and now it just works for me
< instagibbs> oops false alarm, hit or miss
< instagibbs> incognito works reliably here
< jonasschnelli> Indeed. Private browsing mitigates. Strange!
< MarcoFalke> wumpus: minor nit: It says 15.99 on https://dev.visucore.com/bitcoin/doxygen/index.html ; I guess you'd have to run ./configure before generating the docs?
< bitcoin-git> [bitcoin] Empact opened pull request #13019: Trivial: Consistently use FormatStateMessage (master...format-state-message) https://github.com/bitcoin/bitcoin/pull/13019
< bitcoin-git> [bitcoin] Empact opened pull request #13020: Consistently log CValidationState on call failure (master...log-cvalidation-state) https://github.com/bitcoin/bitcoin/pull/13020
< cubancorona> hi, all
< cubancorona> I think signrawtransaction -prextxs should take the json output from decoderawtransaction. Any reason why not?
< bitcoin-git> [bitcoin] jimpo opened pull request #13021: MOVEONLY: Move logging code from util.{h,cpp} to new files. (master...logging-files) https://github.com/bitcoin/bitcoin/pull/13021
< promag> wumpus: friendly ping #12639
< gribble> https://github.com/bitcoin/bitcoin/issues/12639 | Reduce cs_main lock in listunspent by promag · Pull Request #12639 · bitcoin/bitcoin · GitHub
< jamesob> has there been any talk of moving from trusty to xenial on travis? It may be the case that we're unable to run bitcoin-qt using the functional test framework due to a bug in the version of openssl trusty uses
< jamesob> ah, just found #13000 :)
< gribble> https://github.com/bitcoin/bitcoin/issues/13000 | travis: Switch to xenial by MarcoFalke · Pull Request #13000 · bitcoin/bitcoin · GitHub
< MarcoFalke> jamesob: Yeah, they don't have documentation on it, so it is not worth to jump into it too much right now
< luke-jr> jamesob: bionic would be more useful IMO
< luke-jr> problem is vmbuilder (and therefore make-base-vm) doesn't work with it yet :/
< MarcoFalke> luke-jr: I think it makes sense to run on a somewhat older version of gcc, which xenial provides
< MarcoFalke> iirc we don't use vmbuilder on travis?
< bitcoin-git> [bitcoin] Empact closed pull request #13019: validation: Consistently use FormatStateMessage (master...format-state-message) https://github.com/bitcoin/bitcoin/pull/13019
< jamesob> MarcoFalke: I'm happy to pick up 13000 if you don't have time to investigate; looks like you may just need to obtain python's setuptools through a different package (though I'm sure there may be more problems)
< MarcoFalke> jamesob: I think I got it working with that yaml. The issue was that it would just randomly time out without output, not run at all or apt update fails due to locks
< MarcoFalke> Really it is up to travis to make it work first.
< jamesob> MarcoFalke: doesn't look like it: https://travis-ci.org/bitcoin/bitcoin/jobs/367259625
< jamesob> ah but I see other jobs in that build timed out
< bitcoin-git> [bitcoin] jamesob opened pull request #13022: [qa] Attach node index to test_node AssertionError and print messages (master...2018-04-18-func-test-debug-log) https://github.com/bitcoin/bitcoin/pull/13022
< bitcoin-git> [bitcoin] skeees opened pull request #13023: Add unit tests for signals generated by ProcessNewBlock (master...event-tests) https://github.com/bitcoin/bitcoin/pull/13023
< jimpo> jonasshnelli: Odd that processing the validation interface queue would take so long. Are there any other log lines after that?
< jnewbery> promag: I think your #13017 could go on the high-priority for review, since it's a pre-req for dynamic wallet load/create/unload. Agree?
< gribble> https://github.com/bitcoin/bitcoin/issues/13017 | Add wallets management functions by promag · Pull Request #13017 · bitcoin/bitcoin · GitHub
< jnewbery> I've overhauled `loadwallet` and rebased it on that. Ready for review at #10740 if people are interested
< gribble> https://github.com/bitcoin/bitcoin/issues/10740 | [WIP] [wallet] `loadwallet` RPC - load wallet at runtime by jnewbery · Pull Request #10740 · bitcoin/bitcoin · GitHub
< promag> jnewbery: missing postInitProcess on #10740?
< gribble> https://github.com/bitcoin/bitcoin/issues/10740 | [WIP] [wallet] `loadwallet` RPC - load wallet at runtime by jnewbery · Pull Request #10740 · bitcoin/bitcoin · GitHub
< bitcoin-git> [bitcoin] ken2812221 opened pull request #13024: test: Add rpcauth pair that generated by rpcauth (master...rpc_test) https://github.com/bitcoin/bitcoin/pull/13024
< promag> sipa: can you explain IsMineSigVersion::TOP?
< sipa> promag: there's a comment!
< sipa> TOP = 0, //! scriptPubKey execution
< promag> sipa: got it, any reason to not pick BASE?
< sipa> yes, to help review
< sipa> SigVersion::BASE means toplevel or p2sh
< sipa> IsMineSigVersion::TOP means only toplevel, and as it has a different meaning it's better to rename it
< sipa> so that as a reviewer you can easily ascertain that all sites have been addressed
< promag> only top got me confused, toplevel or root would be more clear
< promag> at least for me, as this is not the code I usually see
< sipa> meh :)
< sipa> there's a comment
< promag> yes, and the commit message is also clear about the distinction