<bitcoin-git>
bitcoin/master 32ee767 fanquake: Merge bitcoin/bitcoin#24111: build: force CRCCheck in Windows installer
<bitcoin-git>
bitcoin/master b3ccf26 fanquake: build: force CRCCheck in Windows installer
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
bitcoin-git has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] fanquake merged pull request #24111: build: force CRCCheck in Windows installer (master...force_win_installer_crccheck) https://github.com/bitcoin/bitcoin/pull/24111
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 250 seconds]
morcos has quit [Remote host closed the connection]
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
bitcoin-git has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] MarcoFalke merged pull request #24103: Replace RecursiveMutex `m_cs_chainstate` with Mutex, and rename it (master...m_cs_chainstate_mutex) https://github.com/bitcoin/bitcoin/pull/24103
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
kexkey has quit [Ping timeout: 250 seconds]
kexkey has joined #bitcoin-core-dev
kabaum has quit [Ping timeout: 256 seconds]
kabaum has joined #bitcoin-core-dev
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 256 seconds]
mikehu44_ has joined #bitcoin-core-dev
mikehu44__ has joined #bitcoin-core-dev
mikehu44 has quit [Ping timeout: 256 seconds]
brunoerg has joined #bitcoin-core-dev
mikehu44_ has quit [Ping timeout: 268 seconds]
willcl_ark has quit [Changing host]
willcl_ark has joined #bitcoin-core-dev
mikehu44 has joined #bitcoin-core-dev
mikehu44__ has quit [Ping timeout: 256 seconds]
mikehu44_ has joined #bitcoin-core-dev
mikehu44 has quit [Ping timeout: 250 seconds]
brunoerg has quit [Remote host closed the connection]
brunoerg has joined #bitcoin-core-dev
AaronvanW has quit [Remote host closed the connection]
bitcoin-git has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] PastaPastaPasta opened pull request #24213: refactor: use Span in random.*; make GetRand a template, remove GetRandInt (master...refactor-random) https://github.com/bitcoin/bitcoin/pull/24213
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
brunoerg has quit [Remote host closed the connection]
brunoerg has joined #bitcoin-core-dev
vysn has quit [Quit: WeeChat 3.3]
AaronvanW has joined #bitcoin-core-dev
<provoostenator>
During IBD we download and partially verify blocks out of order, and save them. Moments later, when any gaps are closed, we connect them to the tip.
<provoostenator>
But in this last step, ConnectTip(), we read them again from dis.
<provoostenator>
Wouldn't it be better if we kept those blocks in memory? (if the user wants to and has enough RAM)
<provoostenator>
I find that IBD with -blocksdir pointing to a USB disk is very slow, even with -dbcache set high enough to never need a flush.
<provoostenator>
So I wonder if the / a bottleneck is this writing and then reading.
brunoerg has quit [Remote host closed the connection]
<lightlike>
provoostenator: with a block window of 1024, that would mean an extra ~1.5GB in RAM at current block sizes, right?
<provoostenator>
lightlike: something like that, so it has to be opt-in, similar to -dbcache (which needs 10+GB at the moment)
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
brunoerg has joined #bitcoin-core-dev
AaronvanW has quit [Ping timeout: 256 seconds]
<provoostenator>
That said, running with debug=bench shows the "Load block from disk" step typically takes 20ms which isn't a huge part of the Connect time.
brunoerg has quit [Ping timeout: 268 seconds]
mikehu44_ has joined #bitcoin-core-dev
mikehu44__ has joined #bitcoin-core-dev
mikehu44 has quit [Ping timeout: 256 seconds]
mikehu44_ has quit [Ping timeout: 268 seconds]
BlueMatt has quit [Remote host closed the connection]
BlueMatt has joined #bitcoin-core-dev
pergaminho has joined #bitcoin-core-dev
bitcoin-git has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] fanquake opened pull request #24215: [22.x] guix: ignore additional failing certvalidator test (22.x...backport_guix_darwin_22_x) https://github.com/bitcoin/bitcoin/pull/24215
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
<sdaftuar>
provoostenator: i think bluematt tried this some years ago. if i remember right, it didn't work to produce any kind of speedup. not sure i remember what the investigation turned up, but one thing that comes to mind is that your OS may already be caching filesystem data in memory, so the reads may not be much faster if you do the caching in the application itself.
<jonatack>
#22834 has concept acks by laanwj, luke-jr (merged into bitcoin knots IIUC) and mzumsande, and
<sdaftuar>
it's also possible i'm misremembering or there was some other kind of mistake happening, so if you try it again i'd be curious to know the results
<jonatack>
several ACKs by gleb, myself and prayank, if anyone else would like to weigh in
<jonatack>
also concept ack by practicalswift and tagged for v23.0, so a good time to have a look if you use the onlynet config option
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 268 seconds]
<provoostenator>
sdaftuar: OS-side chacing makes sense. I'm plotting the durations now, and there do seem to be many spikes to 2+ seconds, so I'll investigate that more
<BlueMatt[m]>
IIRC I managed to convince an intern to investigate this and turned up no major performance win, much to my dismay. The hardest part, however, is benchmarking - you have to create some realistic benchmark environment where blocks are downloaded as out of order as they are in real IBD, which isn’t easy.
<BlueMatt[m]>
I still think there’s a win to be hard there, but it’s not clear. OS caching doesn’t explain it, though, because the deserialization is slow, not just the reads. It may be that the increased memory usage causes parts of the leveldb UTXO set to get kicked out of OS cache, though, which could explain it
jespada has quit [Ping timeout: 250 seconds]
<provoostenator>
BlueMatt[m]: right, I'm doing a real IBD on mainnet, with fairly recent blocks. Everything on SSD expect -blocksdir. But it's hard to reproduce the same circumstance
<provoostenator>
Undo data writing also happens in the connect phase.
<provoostenator>
So lots of I/O
<BlueMatt[m]>
To benchmark this I ultimately ended up buying 10 machines with 8GB of memory and thinking I’d go test with those, but I don’t think a super realistic environment was ever built (cc jamesob)
jespada has joined #bitcoin-core-dev
<BlueMatt[m]>
Right, I think you’ll just get wayyyy too much noise with real IBD
AaronvanW has joined #bitcoin-core-dev
vysn has joined #bitcoin-core-dev
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Remote host closed the connection]
<bitcoin-git>
[bitcoin] RandyMcMillan closed pull request #24222: cirrus: add yum clean all to CentOS 8 sequence (master...1643660765-add-yum-clean-all) https://github.com/bitcoin/bitcoin/pull/24222
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
brunoerg has joined #bitcoin-core-dev
Kaizen_Kintsugi_ has quit [Remote host closed the connection]
Kaizen_Kintsugi_ has joined #bitcoin-core-dev
Kaizen_Kintsugi_ has quit [Ping timeout: 250 seconds]
brunoerg has quit [Remote host closed the connection]
Kaizen_Kintsugi_ has joined #bitcoin-core-dev
Guest67tes has joined #bitcoin-core-dev
bitcoin-git has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] theStack opened pull request #24223: test: use MiniWallet for interface_rest.py (master...202201-test-use_MiniWallet_for_interface_rest) https://github.com/bitcoin/bitcoin/pull/24223
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
Guest67tes has quit [Client Quit]
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 268 seconds]
hansland4 has joined #bitcoin-core-dev
brunoerg has joined #bitcoin-core-dev
yanmaani has quit [Ping timeout: 276 seconds]
DeanGuss has quit [Ping timeout: 252 seconds]
DeanGuss has joined #bitcoin-core-dev
DeanGuss has quit [Changing host]
DeanGuss has joined #bitcoin-core-dev
brunoerg has quit [Remote host closed the connection]
yanmaani has joined #bitcoin-core-dev
sipsorcery has quit [Ping timeout: 268 seconds]
Guyver2 has quit [Quit: Going offline, see ya! (www.adiirc.com)]