03:43 <phantomcircuit> sipa: can i just skip abc for blocks during ibd but run checkblock/acceptblock ?

03:52 <phantomcircuit> a priority queue is definitely the right solution, but ... here me out... harder

03:58 <phantomcircuit> hear*

04:18 <phantomcircuit> well actually is it exactly? that's kinda a parallel issue to doing the check/accept block (which writes to disk) in parallel with activatebestchain

05:04 <_aj_> phantomcircuit: doesn't need to be a priority queue, just one queue for blocks and one queue for txs, and you always look at the block queue first?

05:20 <phantomcircuit> _aj_: well it doesn't need to be a priority queue, but it's probably more useful if it is, like certainly we want to prioritize blocks that extend the current tip over ... anything else

05:23 <phantomcircuit> sipa: so how much exactly do we *need* to process a block before handling other messages?

05:41 <_aj_> phantomcircuit: blocks that don't extend the current tip are trivial to process, so don't seem like they should need special casing?

05:44 <phantomcircuit> _aj_: that's not exactly right actually, we do the context independent sanity checks, write the block to disk, and then sometime later read the block from disk when it does extend the tip and... do all those checks again which include calcualteing the merkle root which is quite expensive

05:44 <phantomcircuit> so priority queue could potentially save us doublingg the work we do there

05:56 <_aj_> phantomcircuit: i guess my thought is that AcceptBlock() would be done as-is, and ActivateBestChain() would be the part that's delegated to the thread/queue?

05:58 <phantomcircuit> _aj_: that's what im doing right now, but i think sipa was suggesting going beyond that and putting the expensive to handle messages into a queue

06:00 <_aj_> phantomcircuit: i don't think anything in AcceptBlock is "expensive", except maybe in IBD with assumevalid?

06:01 <_aj_> phantomcircuit: not sure i'm following what your goal is exactly

06:03 <phantomcircuit> _aj_: during ibd the msghand thread ends up pegged at 100% cpu usage which is (obviously) a bottleneck

06:09 <_aj_> phantomcircuit: of which a signifigant fraction is CheckBlock, ContextualCheckBlock duplication in AcceptBlock/ConnectBlock?

06:09 <_aj_> significant

06:10 <_aj_> oh, just CheckBlock, ContextualCheckBlock isn't re-invoked

06:15 <_aj_> phantomcircuit: hmm, no, CheckBlock has an fChecked cache that should early exit?

06:21 <phantomcircuit> _aj_: either way, my goal is to utilitze multiple threads

06:22 <phantomcircuit> a bunch of stuff that's not actually possible

06:27 <phantomcircuit> which isn't going to be trivial here either sinnce activatebestchain holds like... every lock basically

06:28 <phantomcircuit> _aj_: i dont think we write fChecked to disk anywhere?

06:57 <_aj_> phantomcircuit: no, but should be feasible to retain the CBlock with fChecked set while it's in the queue? or to pass through an "assume_already_checked" flag, and set fChecked manually after reading it?

07:38 <bitcoin-git> [bitcoin] maflcko closed pull request #32871: New SVG, Icons, PNGs and X PixMaps (master...master) https://github.com/bitcoin/bitcoin/pull/32871

08:38 <phantomcircuit> _aj_: my entire concept is actually pretty difficult i now realize, i cant write a block to disk without the lock on the chainstatemanager

08:38 <phantomcircuit> which is held by abc

09:06 <_aj_> phantomcircuit: hmm, i would have thought you "should" be able to do that with just cs_LastBlockFile or so. seems like it's all being implicitly guarded by cs_main at present though

09:16 <TheCharlatan> _aj_ that is my impression too.

09:55 <bitcoin-git> [bitcoin] maflcko opened pull request #32874: ci: Catch tests corrupting the source directory (master...2507-ci-less-corrupt) https://github.com/bitcoin/bitcoin/pull/32874

09:59 <_aj_> i suppose you'd need a separate short-lived (reader/writer?) lock for m_block_index for efficient reading ideally.

10:03 <bitcoin-git> [bitcoin] HowHsu opened pull request #32875: index: Fix missing case in the comment in NextSyncBlock() (master...fix-comment) https://github.com/bitcoin/bitcoin/pull/32875

10:19 <phantomcircuit> _aj_: the chainstatemanager lock is just cs_main through a bunnch of indirection

10:20 <phantomcircuit> so im not even gettinng past the ProcessMessage ::BLOCK prev_block lookup when activatebestchain is running

10:21 <phantomcircuit> but now im confused because activatebestchain gives up cs_main specifically to avoid freezing things out for too long?