< BlueMatt>
gmaxwell: yea, i havent had a chance to look into it
< BlueMatt>
midnightmagic: hmm, yea, if thats still possible docs should be fixed to block internet access, not allow it
< dcousens>
phantomcircuit: why would it not be suitable for latency? (not trying to be ignorant)
< dcousens>
for low latency*
< phantomcircuit>
dcousens, because of the exact mechanism ckpool uses
< phantomcircuit>
if blocknotify fails their latency is 30 seconds
< phantomcircuit>
idiots
< phantomcircuit>
i refuse to give them anymore free advise though
< gmaxwell>
04:27 < wumpus> doesn't rust already have libraries for json parsing/generation? or was that mostly an educational exercise?
< gmaxwell>
Needed it for the same reason we need univalue instead of jansson or the like... all other json libraries insist on interperting the values for you.
< Luke-Jr>
well, to be fair, interpreting the values is literally the only thing JSON parsers do.. :p
< phantomcircuit>
Luke-Jr, no they also distinguish between value and object
< phantomcircuit>
although i guess technically that's also "just" a type
< Luke-Jr>
if Rust's JSON lib supported accessing JSON Numbers as the Ratio type, I think that'd be fine
< wumpus>
Luke-Jr: no, parsing is something else than interpreting :)
< Luke-Jr>
not in that context
< wumpus>
anyhow I guess the reason we need this all is the mistake to use JSON for "very large, precise values", apparently this undermines why you'd use JSON in the first place: readily available libraries
< wumpus>
it's a smaller example of why xml is a complex failure, yes in theory it's nice to have an interchange format, but in practice there's always something that makes your use-case slightly different from the general rules, and it may as well break all interoperability :)
< Luke-Jr>
I think everything would mostly just work if we used integer satoshis
< wumpus>
in the end could just as well have used our own, hand-rolled binary packet format, every language needs a special library to interface to bitcoin anyway ...
< wumpus>
Luke-Jr: it may have helped a bit, though some JSON parsers can and do represent integer-looking numbers as doubles as well
< Luke-Jr>
wumpus: doubles can be sufficiently precise in this cae
< Luke-Jr>
case
< wumpus>
in theory, yes, but in practice some platforms's FPUs aren't very nice and precise and introduce subtle rounding errors
< wumpus>
that's the whole problem, at least integer arithmetic is well defined (except for overflow behavior, but we don't get near there for 64 bit)
< wumpus>
Looking for ACKs for #7103 (fix settxfee, paytxfee), this one is quite urgent imo
< dcousens>
wumpus: is there a commonly 'recommended' way to alert/notify node operators if bitcoind has crashed? Its obviously platform dependent and best left up to the operator as to how, but, I think a good 'recommendation' could help people recognize when their node has died if their new to it all etc
< dcousens>
Might be something to include in the docs
< dcousens>
Just a thought
< wumpus>
I'd say that's in the domain of external process-mointoring solutions
< dcousens>
wumpus: absolutely
< dcousens>
But, that can't hurt a 'recommendation'?
< dcousens>
Even a link/reference
< wumpus>
I guess it's be possible to call -alertnotify in the case of a node shutdown
< wumpus>
(at least an unexpected one...)
< dcousens>
wumpus: probably not an assert though?
< dcousens>
or segfault
< wumpus>
but in e.g. a pointer crash that's very difficult
< wumpus>
right
< wumpus>
I'm sure crash handling can be improved in general
< wumpus>
(but it's only one in 23209843 issues that still have to be handled, I can't give it any special priority)
< Luke-Jr>
well, if we cache the command in a C global..
< Luke-Jr>
SIGSEGV could be handled by simply calling exec() :p
< dcousens>
If we don't have an external process-monitoring solution to recommend, then, IMHO, we should provide an option such as `-shutdownnotify` which catches some errors
< wumpus>
the problem is that if something smashed the stack or the heap or etc, trying to do anything may fail or even be an security isue
< wumpus>
it's non-trivial to do
< dcousens>
wumpus: in which case, IMHO, lets just recommend a basic bash script maybe?
< wumpus>
heck, just run bitcoind in a loop in a script and have it send a mail when it quits
< Luke-Jr>
wumpus: even calling exec() with global params?
< dcousens>
wumpus: thats simple to you or I
< wumpus>
this is basic sysadmin 101
< dcousens>
wumpus: also, running it in a loop isn't a good idea in terms of disk space
< dcousens>
could actually be very destructive
< wumpus>
right, then don't run it in a loop
< dcousens>
wumpus: suddenly, software specific edgecases
< dcousens>
(hence, a vetted recommendation is IMHO a good idea)
< wumpus>
see, my ideas about this even suck, let's leave it up to devops professionals instead of trying to handle it crappily in the software
< dcousens>
wumpus: what about in the docs though?
< wumpus>
you can mention it in the docs, of course
< dcousens>
(what I was referring to, agreed on not handling in software)
< wumpus>
maybe what could help is ask around for people that run bitcoind in production on what they're doing - none of my nodes has any uptime requirement and I run them in gdb :)
< wumpus>
hah, already done
< dcousens>
wumpus: haha yeah, also found a mailgun config example :P
< dcousens>
I think that combined with maybe some filesystem alerts etc
< dcousens>
Should be the most useful for people
< dcousens>
(and memory)
< Luke-Jr>
lol
< jonasschnelli>
sipa: does Checkpoints::GuessVerificationProgress require cs_main lock?
< jonasschnelli>
is pindex->nChainTx immutable?
< sipa>
eh... after it has been set, it is
< sipa>
so for things in chainActive, it's immutable
< jonasschnelli>
i don't see a need for locking cs_main during GuessVerificationProgress()
< jonasschnelli>
hmm... chainActive.Tip()?
< jonasschnelli>
that requires a lock i guess.
< jonasschnelli>
but i could pass a CBlockIndex * (received from the signal) there.
< GitHub43>
bitcoin/master cde857f Peter Todd: Connect to Tor hidden services by default...
< GitHub43>
bitcoin/master d6454f6 Wladimir J. van der Laan: Merge pull request #7090...
< GitHub29>
[bitcoin] laanwj closed pull request #7114: util: Don't set strMiscWarning on every exception (master...2015_11_exception_nomiscwarning) https://github.com/bitcoin/bitcoin/pull/7114
< GitHub1>
[bitcoin] jtimon opened pull request #7116: Trivial: Fix warning introduced by #7053 by casting to uint64_t (master...fix-7053) https://github.com/bitcoin/bitcoin/pull/7116
< GitHub146>
[bitcoin] paveljanik opened pull request #7118: Describe maxmempool and mempoolminfee in the getmempoolinfo RPC help. (master...20151127_getmempoolinfo_fixes) https://github.com/bitcoin/bitcoin/pull/7118
< sipa_>
patch-14, merged in 14 minutes. coincidence? i think not!
< paveljanik>
8)
< instagibbs>
Should deprecation warnings be put in debug.log like everything else?
< jtimon>
sipa: should I reopen #6445 now?
< jtimon>
(or an equivalent replacement but a new PR)
< sipa>
jtimon: that's a big refactor, not a small change like 7070
< jtimon>
sipa: so how will I know when "it is time" for #6445 ?
< sipa>
jtimon: that one is part of the move consensus logic to consensus package? i thought you were going to write up a design of what the plan was first
< jtimon>
sipa: the mempool is limited in size and we have rbf, I'm not sure what other "mempool work" I'm waiting not to interfere with
< jtimon>
sipa: that's an optimization, not strictly required for libconsensus but I obviously haven't done it in a way that would hurt my plans
< jtimon>
if that's closed, is for "not interfering with mempool work" just like 6068 and other things
< sipa>
wrt policy encapsulation, my view is that we should only factor things out once they're stable and clearly safe to be configurable
< sipa>
there is no bounds to what policy is
< jtimon>
you mean never?
< sipa>
you can't look at a codebase and say "yes, the policy here is encapsulated!"
< sipa>
ultimately, policy is all of a node's operations that are not consensus and wallet related
< jtimon>
well, I can look at a codebase and say, "no, the policy here is NOT encapsulated"
< sipa>
we're not going to move all of that into a separate package
< sipa>
imho the policy package should just be a few safe and clear things that are configurable
< jtimon>
I'm happy taking slowly it out of mempool, consensus code and main to the same directory
< sipa>
but if there's no clear boundary it's just confusing... you can't guess what belongs where
< sipa>
that's my point, there is no "it"
< jtimon>
well, there' many things are very obvious to me
< sipa>
policy is whatever you make configurable
< jtimon>
so, never, then?
< sipa>
i haven't said that
< sipa>
but i do consider it not a priority, and hard to do now
< gmaxwell>
And that confusion makes code unreviewable... when constantly you can't tell what things do because you have to chase n levels of redirection to find them. If they're configurable, then the answer at review is "anything" and you can review with respect to that.
< jtimon>
I don't think it's hard to start now, not at all, but if I'm waiting for something, I would like to undesrtand better what I'm waiting for
< sipa>
jtimon: wait for having the code that interacts with mempool/relay to be stable, wait a release, and propose a plan for things to become policy
< jtimon>
so 6445 and 6423, for example, make the code more confusing?
< sipa>
now it's just random moves, and (other than with consensus code) there is no obvious way for what is policy and what isn't
< jtimon>
sipa: another doc for policy?
< jtimon>
I couldn't do that, as you say, that potentially never ends (unless there's ever a "right place" for people to put new policy stuff)
< sipa>
6445 is not policy encapsulation
< jtimon>
no, but it was said (I can find the comment) that it would interfere with mempool code
< sipa>
it's just churn...
< jtimon>
that's why I closed it (that's what it has in common with 6068)
< gmaxwell>
jtimon: I think you're missing what sipa is saying; everything that isn't consensus or wallet is policy, in an absolute sense. So unless the 'policy encapsulation' is just renaming main.cpp to policy then it is doing something more specific than encapsulating policy. Sipa suggests a candidate defintion for a narrower definition of policy which is that policy is things which are configurable.
< jtimon>
gmaxwell, there's many more things than relay policy
< gmaxwell>
I did not refer to relay.
< jtimon>
I did
< jtimon>
basically what I undesrtand by policy is relay and block creation policy
< gmaxwell>
Even this is inexorably tied into the rest of the code. E.g. consider my most recent pages, none of them change constants but each of them substantially changes what a node will relay.
< sipa>
jtimon: i really wish you'd work more on consensus encapsulation, write up where you think what should end up, have a plan, get people to look at it, and then at once move everything... i would love to see all consensus code encapsulated away in at least a separate directory... it will take some time to get people to agree, but i think it's something that needs to happen
< jtimon>
anyway, I think I got my answer the "right time" for the policy stuff is "when code that interacts with mempool/relay to be stable", that is, never
< sipa>
jtimon: instead of constantly open pull requests that move unrelated things, close them, rebase them, burden reviewers with change that interferes with things that need to happen now
< sipa>
jtimon: it was stable for a long time before the mempool limiting and all related changes happened
< jtimon>
sipa: I know what I will do about consensus already, I'm just trying to decide what to do with policy
< jtimon>
well, I was trying to make some of those changes before that happen to...
< sipa>
i'm aware
< jtimon>
anyway, I'll rewrite my policy stuff on bitcoin jt 0.12, just like my consensus stuff and then decide, worse case I can just rebase a giant branch once per major release
< sipa>
i'd wish you'd focus on consensus encapsulation
< jtimon>
sipa: I'm sorry to get so bored about writing that pdf, but no hurry, right? no consensus refactors after 0.12 anyway...
< jtimon>
in fact the document kind of depends on the outcome of #7091
< jtimon>
fwiw, I'm not coding anything policy-related
< GitHub102>
[bitcoin] petertodd opened pull request #7119: Add option to opt into full-RBF when sending funds (master...2015-11-opt-into-full-rbf-option) https://github.com/bitcoin/bitcoin/pull/7119
< gmaxwell>
"subver": "/bitcoinj:0.14-SNAPSHOT/" sending me 38 pings in a second. :-/
< sipa>
maybe we need some exponentially decaying badness score, and increment for everything peers do
< gmaxwell>
and decrement for a few things that are decidely good.
< sipa>
like being the first to relay a valid block?
< gmaxwell>
yes, or a transaction we accepted.
< gmaxwell>
Then this score could just be another node sort criteria in eviction/rotation.
< sipa>
yeah
< sipa>
and if gets actually high, it could turn into ban
< sipa>
also, i think banscores should persist across reconnects
< gmaxwell>
e.g. have the initial score be 0, with an IIR decay to 0. Add score for every byte recieved, substract score for every byte in a getdata we asked for; or ping response we initiated.
< gmaxwell>
strikes me as kind of risky that CValidationState is initilized to MODE_VALID in the constructor.
< aj>
gmaxwell: (maybe make it a goodness score instead, so positive values are, well, positive?)
< gmaxwell>
aj: envision me looking crossly at your shed painting.
< aj>
gmaxwell: (isn't it pretty!)
< aj>
gmaxwell: (it can store scooters as well!)
< PRab>
sipa gmaxwell: Would a persistent banscore make mitm attacks more powerful because p2p comms are unauthenticated right now?
< Taek>
on a similar note, if you decrement for things that are good, an attacker can leverage that to intentionally get ahead. Unless your average peer has a zero-badness score in the first place
< gmaxwell>
Taek: they can't if being good is inherently limited.
< gmaxwell>
(except by actually being good, in which case they're not much of an attacker!)