<@wumpus>
you're not the first to do this but probably the first to try to formalize the data format. There's also some synergies with e.g. the zmq notification protocol here.
< aj>
kallewoof: fwiw, i have about 12GB of compressed mempool tx data from the last few months, with the idea to some day analyse if any of the non-confirming txes display any interesting behaviour
< aj>
kallewoof: the 1-satoshi-is-plenty phase started shortly after i started keeping track though, so it's probably boring :(
<@wumpus>
after all, a log format is pretty much *concatanate all the notifications*
< aj>
kallewoof: (my data is basically hexified zmq data, with a timestamp, then lz compressed)
< aj>
kallewoof: includeconf travis -- test_runner is failing travis of the warning stderr output i think?
< kallewoof>
aj: Oh, you have data? I would love to get my hands on it!
< kallewoof>
wumpus: I figured, yeah. Having a formalized format would be beneficial I think, unless the format is crap. (Hence RFC :) )
<@wumpus>
kallewoof: yes
< kallewoof>
aj: Ah... of course. I'll switch to regular printf()..
< kallewoof>
wumpus: I should def look at zmq notification protocol, yeah. I forgot about that.
<@wumpus>
kallewoof: but you have a good point I think the difference conceptually is that the zmq notification protocol is specific to bitcoin core, and includes some specific implementation details, while you're trying to make a more general exchange format
<@wumpus>
(e.g. we have many more mempool retirement reasons, but some of those are very specific)
< fanquake>
kallewoof Followed static analyzer your steps, seems to be working correctly. Cheers.
< kallewoof>
wumpus: What other than 'old not confirmed', 'rbf-replaced', and 'double-spent-in-block'?
< kallewoof>
fanquake: Cool :)
<@wumpus>
kallewoof: see MemPoolRemovalReason enum
< aj>
kallewoof: most of the data's behind my home internet which will make getting it out usefully less than easy :)
< kallewoof>
aj: Nice. Downloading
< kallewoof>
aj: Would access to an ssh machine somewhere be helpful?
< kallewoof>
s/ssh machine/ssh account/
< aj>
kallewoof: maybe? somewhere with disk to put it would be helpful; otherwise it's just upstream-b/w-bound
< kallewoof>
aj: Wasn't sure if your problem is bandwidth or simply don't have a disk online with enough space. I can give you access to a digital ocean machine somewhere.
< kallewoof>
aj: I got the files. Thanks, will check soon :)
< aj>
kallewoof: well, probably be more useful to see if the data's actually interesting; b/w and storage i can solve myself without too much hassle :)
< kallewoof>
aj: Makes sense yeah
< jonasschnelli>
If someone is interested, one of my machines does dump the mempool (mammempool 300MB) every hour since a couple of month... a lot of data though
<@wumpus>
the incremental format is probably slightly more efficient in storage, sounds like dumping the entire thing every hour will get you a lot of duplicate data, though compression helps ofc
< jonasschnelli>
Yes! I had to trade off storage space vs. implementing an incremental dump solution
<@wumpus>
yep
< fanquake>
jonasschnelli Thanks
< jonasschnelli>
fanquake: I need to finish my IRC bot that allows triggering a gitian build on my machine
< jonasschnelli>
Building all pull requests would require a couple of machines (=much more expensive setup)
<@wumpus>
indeed, I think it's better to have it as something manually triggered, at least at first, can always be automatized if there's need for it
< jonasschnelli>
Also, there are risks running pre-built binaries...
<@wumpus>
yes, there are security issues with building ,as well as hosting binaries for PRs that no human has looked at
< fanquake>
handy way for someone to submit a malicious PR and get their binaries built ready for distribution
<@wumpus>
exactly
<@wumpus>
it does leave one hell of an audit trail, so I doubt it's the best way to parasitize on someone else's infrastructure to host binaries, but still we really want to avoid it
< jonasschnelli>
Trigger a build could be protected over a github username (via special comment) or via an IRC handle. Both not very secure... additionally a gpg signature coule be added
<@wumpus>
jonasschnelli: well as long as you check that freenode user is authenticated and using TLS, your only worry is that freenode itself is compromised, it's unlikely anyone will go that far to trigger a build :-)
< jonasschnelli>
heh, true.
<@wumpus>
same for spoofing github handles I think
< fanquake>
Probably easier to spin up some EC2 instances using credit cards before compromising the freenode infrastructure
< fanquake>
*stolen credit cards
< fanquake>
wumpus Did you want to merge #12715 tomorrow or the day after? I've just about finished testing, looks like it's ready to go. Just want to minimise disruption.
<@wumpus>
<jonasschnelli> fanquake@testing is on fire! <- yeah, thanks for all the testing!
<@wumpus>
fanquake: will take a look
<@wumpus>
fanquake: I was leaving that to marcofalke, but I guess, as we all seem to agree we can just close it, someone who needs the patch right now could always pull that branch I guess...
< bitcoin-git>
[bitcoin] Empact opened pull request #13019: Trivial: Consistently use FormatStateMessage (master...format-state-message) https://github.com/bitcoin/bitcoin/pull/13019
< bitcoin-git>
[bitcoin] Empact opened pull request #13020: Consistently log CValidationState on call failure (master...log-cvalidation-state) https://github.com/bitcoin/bitcoin/pull/13020
< cubancorona>
hi, all
< cubancorona>
I think signrawtransaction -prextxs should take the json output from decoderawtransaction. Any reason why not?
< bitcoin-git>
[bitcoin] jimpo opened pull request #13021: MOVEONLY: Move logging code from util.{h,cpp} to new files. (master...logging-files) https://github.com/bitcoin/bitcoin/pull/13021
< jamesob>
has there been any talk of moving from trusty to xenial on travis? It may be the case that we're unable to run bitcoin-qt using the functional test framework due to a bug in the version of openssl trusty uses
< jamesob>
MarcoFalke: I'm happy to pick up 13000 if you don't have time to investigate; looks like you may just need to obtain python's setuptools through a different package (though I'm sure there may be more problems)
< MarcoFalke>
jamesob: I think I got it working with that yaml. The issue was that it would just randomly time out without output, not run at all or apt update fails due to locks
< MarcoFalke>
Really it is up to travis to make it work first.
< jamesob>
ah but I see other jobs in that build timed out
< bitcoin-git>
[bitcoin] jamesob opened pull request #13022: [qa] Attach node index to test_node AssertionError and print messages (master...2018-04-18-func-test-debug-log) https://github.com/bitcoin/bitcoin/pull/13022
< bitcoin-git>
[bitcoin] skeees opened pull request #13023: Add unit tests for signals generated by ProcessNewBlock (master...event-tests) https://github.com/bitcoin/bitcoin/pull/13023
< jimpo>
jonasshnelli: Odd that processing the validation interface queue would take so long. Are there any other log lines after that?
< jnewbery>
promag: I think your #13017 could go on the high-priority for review, since it's a pre-req for dynamic wallet load/create/unload. Agree?