< morcos> jamesob: i'm also working on CNB improvements right now
< morcos> however the background threading was further down my list, so perhaps we can coordinate our work to some degree
< morcos> right now i'm refactoring CNB into a class (i should have just done that last time)
< morcos> i'd be interested in understanding exactly what you're trying to accomplish though, b/c it's not really clear to me how to gain a whole lot without much more locking rework.
< morcos> certainly there are a couple of good ideas that should be implemented and which we've seen PR's take a stab at though:
< morcos> 1) moving TestBlockValidity to a background thread (although its not clear how easy it is to make this interuptable and this is 90% of the time of CNB right now)
< morcos> 2) returning a template based on no txs after a new tip
< sipa> making CNB as a whole run in the background would make taking TBV out of the delay very easy: just set the stored CNB result before verifying it
< morcos> 3) interrupting the cs_main lock required for block assembly (before TBV) when a new tip comes in, although now this is only 10-20ms and it really shouldn't require so much cs_main as a mempool lock which just happens to be cs_main now
< morcos> sipa: i suppose that's true.
< morcos> i guess it just seems to me that its almost more logical to let the caller dictate when (how often) the new block is assembled
< morcos> the latency is irrelvant except when a new tip comes in, and thats really the interupt function you need to get right
< morcos> and then can work just as well or even better in the same thread
< morcos> longpoll returns after a) some specified amount of time has passed, and then at the end of it a new template is created or b) a new tip comes in
< morcos> i mean it really already is a "background" thread
< morcos> you just want to return the result before running TestBlockValidity, and you want to have an interupt for new tips
< morcos> Or am I missing that there is some advantage of immediately returning a template that is Y seconds old when you call gbt instead of having to wait X seconds for a template that is X seconds old where X is by definition less than Y b/c X is the block assembly time
< morcos> sipa: do you have any thoughts on the feasibility of making ConnectBlock interuptable? That seems to me its always going to be an issue with generating a lot of different blocks if you want to keep testing them.
< BlueMatt> morcos: please do not further complicate getblocktemplate
< BlueMatt> we want to move complexity out of pool servers and into bitcoind, not the other way around
< BlueMatt> (pool servers are largely bit proprietary blobs that get essentially no external review, whereas bitcoin core is....not that)
< sipa> morcos: i'd rather just not always call TBV... *ducks*
< BlueMatt> sipa: noooooooooooooooooo
< BlueMatt> that shit has been so broken up until literally the latest release
< BlueMatt> there is no way in hell we should remove that check
< sipa> i am not saying remove it
< sipa> but just call it 1% or 10% of the time or so
< sipa> it's there to detect software bugs
< sipa> it's not there to prevent incorrect responses
< BlueMatt> sipa: it is absolutely there to prevent incorrect responses
< BlueMatt> up until the latest release there were known bugs which made it required
< BlueMatt> (well assuming you dont check txn as you go, which we now dont)
< sipa> ... and cause the software to crash?
< sipa> it's an assert
< BlueMatt> it isnt anymore
< BlueMatt> morcos: in any case, my view for getblocktemplate refactors, which should be a few small changes (that I suggested to jamesob last night) is
< BlueMatt> 1) Interrupt CNB when a new block comes in...you suggest block validation times are not an issue, but, really, pools do not run with our fancy hardware or mempool limiting (hell, when talking about mempool limiting we often assumed miners have a massive mempool)
< BlueMatt> 2) run CNB in the background on a loop...making getblocktemplate never block. Often when pools call getblocktemplate, they do it very often, and you'll end up blocking accepting a new block for getblocktemplate to return.
< BlueMatt> 3) return an empty block if the current block template we have is not based on the tip
< BlueMatt> 3 should be a trivial change, and 1 and 2 should be pretty easily-reviewable code
< BlueMatt> between the 3 getblocktemplate is about as optimal as we need to care, and all of the questionable logic currently in pools to emulate this can go away
< sipa> 3) means that the update new tip code should also briefly lock the CNB cached value and wipe it
< BlueMatt> morcos: which looks almost identical to your proposal, but I dont think we should only be moving TBV to a background thread
< BlueMatt> all of CNB should be there, getblocktemplate should never block
< sipa> otherwise GBT needs to grab cs_main to determine whether the tip changed
< sipa> (not a problem; just realizing this)
< BlueMatt> sipa: good point
< morcos> ok hold on catching up on scrollback
< morcos> BlueMatt: ok first of all i'm not suggesting moving more logic into pool servers. i 100% agree that bitcoind should on its own provide near optimal mining functionality
< BlueMatt> morcos: note that I only realized how similar our proposals were after going back and reading what I wrote :)
< morcos> for your #1, what do you want to interrupt? 90% of CNB is TBV which is mostly ConnectBlock, which is why I asked sipa about interrupting it?
< morcos> I'm not necessarily opposed to moving all of CNB into a separate thread, but I have two concerns with it, both minor'ish..
< BlueMatt> honestly I'd love to re-add the during CNB txn validity checks that were removed for 0.12, but I'll leave that one for now...in any case, in this case ConnectBlock only touches its own datastructures, so making it return when a particular flag gets set should be doable
< BlueMatt> annoying to shove that in ConnectBlock, but, doable
< BlueMatt> we could do more fucking thread::interruption_points
< morcos> a) if CNB still acquires cs_main for a lot of time, then busy running it is going to block a lot of other stuff even if somehow new tips can interrupt it, and thats not useful if the mining software isn't getting the new blocks that quickly. you only need as many new blocks as often as they are going to switch work
< morcos> b) when you do return work to the miner, its sort of silly to return work that is old by the period of your CNB loop instead of returning it on demand
< BlueMatt> hmm? I'm confused on a) are you just talking about the other non-new-block shit that locks cs_main?
< morcos> yes
< morcos> a lot of the nodes operation does this
< morcos> acceptToMemoryPool being among the worst offenders
< BlueMatt> yea, I'm not too worried about that....mostly because the rest of that stuff is very non-latency-critical
< sipa> accepting new blocks is
< morcos> BlueMatt: as for your comment about massive mempools or fancy hardware. CreateNewBlock speed is not related to mempool size, or only just barely.
< BlueMatt> yea, aside from a new block, nothing is latency-critical
< BlueMatt> morcos: true, I suppose that is now true...still thinking in a pre-0.12 world :)
< BlueMatt> morcos: oops, no, you misread my comment, I was suggesting accepting a new block is related to mempool size
< BlueMatt> which it still is, if you have a completely insanely large mempool
< sipa> how so?
< morcos> as for adding back in the tx validity checks. one of the reasons i was happy to take them out is it would have been just as easy to have an error in how those were applied as have an error in mempool consistency
< BlueMatt> that plus disk i/o and cache size, and some miners have surprisingly low cache/disk throughput
< BlueMatt> morcos: in theory, sure, but we've never had a release where mempool was consistent (and we thought it was for many, many releases), but we've never found any issues in the way the txn checks were done in CNB
< morcos> in fact we do have an error in how they are applied already. 0.11 will just has happily build a block that fails TBV as 0.12 when a new soft fork activates
< BlueMatt> going back to your (b) above, meh, I'd much, much rather do that than block and make miners do some kind of crazy hacky solution because of it
< BlueMatt> morcos: oh? do we really? fuck :(
< sipa> how so?
< morcos> BlueMatt: back to b a second. I'm confused, what are minors doing that you are calling a crazy hacky solution
< sipa> we never accept transactions that violate a future softfork into the mempool
< sipa> even a non-activated one
< BlueMatt> morcos: one major miner who happens to have a reasonable infrastructure has one pool server (with some failover stuff) controlling their hardware....connected to about 5/10 bitcoinds on the backend
< morcos> sipa: oh yeah sorry, i think i meant if you're running wiht non default policy
< BlueMatt> morcos: you have to do that for latency of accepting a new block reasons
< BlueMatt> morcos: but, really, now that I say that, it doesnt really effect what we're talking about :)
< morcos> BlueMatt: exactly. once you do your number 3, then there is no more benefit to be gained beyond moving TBV out of thread
< BlueMatt> in any case, my thought is really that if you block gbt, pool server implementors are going to have to build smarter software that does things like threading or calling select() instead of being able to be just as effecient as the next guy by just using blocking sockets :)
< morcos> but at this point we're talking micro optimizations. kind of depedns on how often minors want new work, and how frequently we think running CNB is reasonable.
< sipa> you'd certainly not run CNB in a busy loop
< BlueMatt> yea, we should have a discussion about how often we re-call CNB later
< sipa> that'd interfere greatly with other cs_main lockers
< BlueMatt> probably something related to how many new txn you have in mempool or so
< BlueMatt> but...first lets get the easy stuff in
< morcos> BlueMatt: thats already how it works
< BlueMatt> i thought it currently has a timeout?
< BlueMatt> meh, i havent looked at that code in too long, i guess
< morcos> BlueMatt: yeah maybe it waits the timeout, and then also requires that the # of txs changed
< morcos> but the point is it already has logic like that, tweakign the logic is easy
< BlueMatt> in any case, my thought is really that it is easier to write a pool if calls to gbt never take longer than an ms or two than if they sometimes block
< morcos> but anyway, sounds liek we agree on what we're trying to accomplish
< BlueMatt> really my goal is to make a braindead pool software just as fast as a really smart and overengineered one
< morcos> BlueMatt: ok, well if there is a difference between blocking for 10ms and blocking for 100ms then i agree with you
< morcos> you could assemble the JSON response ahead of time too
< BlueMatt> I mean fuck if I know, but if I'm writing something really braindead, if it blocks for 100ms I have to actually think about what to do, but if its 10 or 1, then I can just do whatever I think of first
< BlueMatt> heh, could do that, too
< morcos> but we should figure out how to interrupt connectblock
< BlueMatt> yea, that sounds painful
< sipa> how about we ship Varnish along with bitcoind to cache the GBT results *ducks*
< morcos> also, we should have connetblock be able to give up its cs_main after connecting inputs and before validating sigs
< BlueMatt> but we can since jamesob was asking for simple tasks, I was trying to avoid scope creep at all costs
< BlueMatt> morcos: yea, that
< sipa> morcos: yea, that
< morcos> hmm... so for the simple first version
< morcos> 1) empty block template, all agree on that
< morcos> 2) need to not wait on TBV at least. if we move all of CNB into its own thread... does it just run on a schedule continuously. is that close enough to ok... i guess maybe so..
< BlueMatt> morcos: we used to be able to skip cs_main for the ConnectBlock call at the end to validate because we had a CCoinsViewCache that was filled with txn previns when we ran through and did the validation on each txn
< BlueMatt> now we cant so easily :(
< BlueMatt> (because ConnectBlock's calls to the CCoinsViewCache fall through to pcoinsViewTip since the cache is empty when we start checking validity)
< BlueMatt> oh, nvm
< morcos> one thing i don't like the idea of is waiting say 10 seconds to give a second template with actual txs in it after a new tip
< morcos> if you have that information at T+10ms, it seems crazy to spend 10secs or 30 or 60 building empty blocks
< BlueMatt> <BlueMatt> oh, nvm <-- nope, my previous statements was correct, ignore this comment
< morcos> we have to make sure we get that switchover right
< morcos> yeah i guess this is where i don't understand what pool server software should look like
< morcos> the longpoll idea makes sense to me... b/c bitcoind tells you when hey, here is new info you need
< BlueMatt> agreed
< morcos> vs you having to know when to ask
< morcos> so its by definition blocking for a while
< BlueMatt> I dont think most pool servers use longpolling, because it is often slower or something
< BlueMatt> i dont know why, havent looked into it
< morcos> well thats the question we need to answer
< BlueMatt> as for the switchover, another option is just to tickle the background thread when we realize the block is now out of date
< BlueMatt> lose an extra ms on switching time, but thats fine
< morcos> b/c it seems simpler to me to write braind dead pool software that long polls than pool software that has to figure out itself when it should ask bitcoind for info
< morcos> yes but how does the pool know to ask again
< morcos> is it literally going to busy call gbt and check if it changed?
< BlueMatt> some software does this, yes
< btcdrak> morcos: I dont think jamesob saw your messages, says he quit before you replied.
< aj> btcdrak: sorry, i got distracted turning my htlc code into something closer to a working demo; should be up on github today or tomorrow
< btcdrak> aj: thanks!
< Luke-Jr> jonasschnelli: meh, may need to find another font still :x
< Luke-Jr> jonasschnelli: reason for generating a custom DS_Store, is that it needs the volume name in it
< Luke-Jr> fwiw, gentoo testing (not stable) just got GCC 5.3.0
< Luke-Jr> found a half-decent public domain font we can use
< Luke-Jr> jonasschnelli: can you minimally modify the generated DS_Store to work, and post me the working file?
< Luke-Jr> jonasschnelli: (not on your broken SSL server..)
< Luke-Jr> hm, maybe I found a trivial bug in RBF logic.. it seems to be checking the replacement pays more fees than the replaced *plus* what would be required of the replacement on its own
< Luke-Jr> rather than what would be required for both..
< Luke-Jr> so if the replaced transaction had 1 BTC in fees, the replacment must have 1 BTC + relayMinimums(replacement), when it might be argued that it ought be relayMinimums(replaced) + relayMinimums(replacement)
< Luke-Jr> thoughts?
< aj> Luke-Jr: the 1BTC already includes relayMinimums(replaced) though?
< MarcoFalke> But relayMinimums(replaced) + relayMinimums(replacement) can be smaller than 1 BTC
< Luke-Jr> ^
< Luke-Jr> so what we really would want is fee(replacement) > max(relayMinimums(replaced)+relayMinimums(replacement), fee(replaced))
< aj> yeah, i thought that was to make RBF deliberately "expensive"
< Luke-Jr> I don't follow..
< Luke-Jr> unless you mean logic-less inflation of RBF costs
< aj> i've seen gmaxwell (iirc) explain it on reddit as RBF having to pay for the cost of replacing both transactions
< aj> i didn't follow why that was a good thing though
< Luke-Jr> oh
< Luke-Jr> if we use rM(repl'd)+rM(repl), we get a DoS attack because they could both be exactly 1 BTC and replace each other
< Luke-Jr> we'd need a +1
< MarcoFalke> It's max(relayMinimums(replaced)+relayMinimums(replacement), fee(replaced) + relayMinimums(replacement))
< MarcoFalke> which is the same as 1 BTC + relayMinimums(replacement)
< MarcoFalke> and you already explained why
< Luke-Jr> I guess max doesn't do what I mean
< Luke-Jr> wait, yes it does
< Luke-Jr> MarcoFalke: it should be max(relayMinimums(replaced)+relayMinimums(replacement), fee(replaced) + 1), no?
< MarcoFalke> what unit is "1"
< MarcoFalke> satoshis?
< Luke-Jr> sure
< Luke-Jr> it just needs to be > to prevent recursion
< GitHub32> [bitcoin] MarcoFalke opened pull request #7218: [qt] Fix misleading translation (master...MarcoFalke-2015-trivial7) https://github.com/bitcoin/bitcoin/pull/7218
< MarcoFalke> Then you can just spend 1 satoshi to transmit whatever your tx size is to the whole network. Sounds like DOS?
< Luke-Jr> MarcoFalke: only if you already paid more than enough in fees
< Luke-Jr> ie, if you overpaid on your initial tx already, you can pay only 1 sat. more for replacement
< aj> Luke-Jr: 2*relayMinimum could still be well below what's being accepted into blocks though?
< Luke-Jr> aj: I don't see relevance.
< jonasschnelli> Luke-Jr: IIRC, the .DS_Store does link to the filename,.. so, if the filename (App Name) is dynamic, the DS_Store files need also. So you need a script that can generate/modify a DS_Store file.
< Luke-Jr> jonasschnelli: yes, but I am lost for what the problem is, so analysing a fixed file would possibly help
< jonasschnelli> Luke-Jr: SSL. I testes the Site with Ubuntu 12.04 and 15.04 as well as with a recent Fedora version. I guess the COMODO SSL CA is probably not supported by your OS?
< Luke-Jr> jonasschnelli: it's not a cert error
< jonasschnelli> Luke-Jr: Okay. Right. Makes sense. Will try to create a correct DS_Store
< jonasschnelli> Does your browser/os enforce SSLv2 or v3?
< GitHub128> [bitcoin] luke-jr opened pull request #7219: Make RBF policies optional (master...rbf_opts) https://github.com/bitcoin/bitcoin/pull/7219
< GitHub143> [bitcoin] luke-jr opened pull request #7220: RBF: Allow replacements to pay for minRelayFee(replaced)+minRelayFee(replacement) rather than actualFee(replaced)+minRelayFee(replacement) (master...rbf_feecomp) https://github.com/bitcoin/bitcoin/pull/7220
< Luke-Jr> jonasschnelli: probably?
< * Luke-Jr> ponders if wget would work
< jonasschnelli> Luke-Jr: give it a try with `curl -v` (and maybe a `curl -v --insecure`..
< Luke-Jr> hm, got a link handy?
< Luke-Jr> for some reason I am having trouble finding one in my log
< Luke-Jr> nope, curl and wget hate it too
< jonasschnelli> argh. Some system wide problem...
< Luke-Jr> I don't think curl does SSL at all, just TLS
< Luke-Jr> ooh
< Luke-Jr> jonasschnelli: IPv6!
< Luke-Jr> you're only broken on IPv6, not IPv4
< jonasschnelli> Luke-Jr: hmm.. how could
< Luke-Jr> is the IPv6 address correct? ;)
< Luke-Jr> in DNS
< jonasschnelli> buh... no idea... need to check. :)
< Luke-Jr> if not, I'm hitting some random other server :P
< * jonasschnelli> is reading some IPv6 guides...
< jonasschnelli> Luke-Jr: hmm... IPv6 seems to be set up coorect. dig AAAA bitcoin.jonasschnelli.ch gives me the right answer, .. i can ping6 from another machine (in the same lan although).
< Luke-Jr> I can ping it, just TLS fails
< jonasschnelli> ah... found the issue. Missing apache ipv6 config.
< Luke-Jr> XD
< Luke-Jr> works now
< jonasschnelli> Luke-Jr: works? Perfect!
< Luke-Jr> jonasschnelli: Our scheme looks more (not everywhere) after full length parameters/commands (we use createrawtransaction instead of createrawtx, etc.). <-- more what? O.o
< jonasschnelli> Luke-Jr: things like '-limitfreerelay', or -stopafterblockimport or -permitbaremultisig
< jonasschnelli> But right, there are also things like: acceptnonstdtxn
< Luke-Jr> more examples don't help; the sentence is missing an adjective :p
< jonasschnelli> but replacebyfee is relatively short.
< Luke-Jr> "Our scheme looks more after full length … " doesn't parse <.<
< jonasschnelli> Don't judge my english..,. (or i gonna judge your german). But I know, my english lacks here and there. :)
< Luke-Jr> ok, but I don't know what it means :<
< jonasschnelli> I was trying to say, our command line arguments and rpc commands are mostly "full names" instead of abbreviations.
< Luke-Jr> ah
< GitHub62> [bitcoin] parazyd opened pull request #7221: Merge pull request #1 from bitcoin/master (master...master) https://github.com/bitcoin/bitcoin/pull/7221
< GitHub110> [bitcoin] parazyd closed pull request #7221: Merge pull request #1 from bitcoin/master (master...master) https://github.com/bitcoin/bitcoin/pull/7221
< MarcoFalke> > Luke-Jr
< MarcoFalke> MarcoFalke: only if you already paid more than enough in fees
< MarcoFalke> For one transaction, that is.
< MarcoFalke> But the network has to process every rbf tx
< MarcoFalke> Just think about the "strss-test" in August
< MarcoFalke> rbf with your logic would make it a lot cheaper
< MarcoFalke> jonasschnelli, theoretically you could also change gui settings without the gui
< jonasschnelli> MarcoFalke: how?
< gijensen> PR 7220 Is it heavy to consider all TXs that were replaced in the chain up until the first TX with an RBF flag that didn't get replaced? That's the only solution I see to morco's NACK
< MarcoFalke> In the registry and stuff, but "GUI settings" is not true as well. I will just use your suggestion.
< jonasschnelli> MarcoFalke: Hm... sure, you can change everything by accessing the right space on your OS. I still think "GUI settings" is the right name.
< helo> someone familiar with p2pool code may want to take a look at KipIngram's messages in #bitcoin
< GitHub4> [bitcoin] luke-jr closed pull request #7220: RBF: Allow replacements to pay for minRelayFee(replaced)+minRelayFee(replacement) rather than actualFee(replaced)+minRelayFee(replacement) (master...rbf_feecomp) https://github.com/bitcoin/bitcoin/pull/7220
< morcos> Am I allowed to store a reference to chainparams?
< morcos> In particular refactoring CreateNewBlock to be a class needs access to chainparams, can i just pass it in as a reference on construction and access it whenever after that point?
< sipa> imho yes
< GitHub68> [bitcoin] sdaftuar opened pull request #7222: [WIP] RPC: Indicate which transactions are signaling opt-in RBF (master...add-optin-info) https://github.com/bitcoin/bitcoin/pull/7222
< gijensen> Luke-jr, PR 7220 for my sanity, you closed it because it doesn't cache the minimums (so the proper min can't be calculated)? That's what I thought but two other people disagreed with me :S
< Luke-Jr> gijensen: ?
< gijensen> Luke-Jr I'm just asking for clarity on the comments previous to your final one. I was trying to convey the issue is the PR as is didn't account for all the minimum fees of past transactions. Is that correct, or no?
< gijensen> You mentioned a cache, so that made me believe I'm correct. I just want to know to be sure I properly understand what's happening.
< Luke-Jr> yes, if A is replaced by B, and C might replace B, we need C to pay for A+B+C, but we no longer have A
< gijensen> Thank you!
< Luke-Jr> so to fix this, we would need to store the size of A (and anything it replaced) with B
< Luke-Jr> nTotalRelaySize or smth
< gijensen> Okay, yes I understand completely. Do you still plan on trying to work on the concept? Or just axe it for now?
< Luke-Jr> I think the benefits do not outweigh the costs now
< gijensen> I was worried about that, okay