<fanquake>
Seems like the wrong two questions? Shouldn't it just be, "What compiler do you use to generate code coverage?".
kevkevin has joined #bitcoin-core-dev
<janb84>
clang + llvm-cov
<vasild>
Well, if we want to know how many people would be affected if we drop X, then the question to ask it "Who is using X?". Also, another thing to consider is that people that use GCC but did not generate coverage in the past are potential users of the to-be-dropped stuff.
<fanquake>
Yea. All of our fuzzing infra, mutation testing etc all use Clang, because it's the only thing that works
<maflcko>
for one-off coverage I just use assert(false) or std::cout and see if it is hit (works with any compiler without coverage tooling)
<fanquake>
Those people can still do whatever they want, assuming they can make it work amongst the matrix of tooling and lcov versions and gcov versions (another reason why it doesn't work out of the box consistently)
<fanquake>
My suggestion is just that we should switch what we have implemented, to be more convenient, for the majority of people already doing useful work (a few exceptions obviously, but that can likely accomodate/don't care too much)
<fanquake>
I'm not saying we shouldn't remove it, I'm saying we implement it so it works with Clang, rather than GCC
<fanquake>
*should
<hebasto>
Using clang for coverage reports doesn’t need a dedicated build configuration at all
<fanquake>
(and not try and use any compatibility modes etc)
<fanquake>
Sure, but neither does GCC
antanst9 has quit [Ping timeout: 272 seconds]
kevkevin has quit [Ping timeout: 244 seconds]
<hebasto>
the coverage build configuration facilitates helping scripts when using cmake multi-config generators.
antanst9 has joined #bitcoin-core-dev
<_aj_>
i've done a cov build in the past; don't remember if it was gcc or clang. i mostly found the upstream tools confusing, vs problems with the autotools/make stuff (which is why i haven't tried it recently). my config.log suggests i was trying with clang at one point for whatever that's worth
<fanquake>
Given multi-config is primarily to support Windows, it seems less relevant here? Is anyone doing multi-config converage builds (on windows?)?
<fanquake>
_aj_: if you tried using Clang with the current coverage build type, you likely would have run into issues yea
<_aj_>
fanquake: i mean "how do i get useful reports? ah, this then this then that then the other thing. ugh, it worked, but that was so annoying i never want to do it again" from what i remember, anyway
<_aj_>
fanquake: easier to use the drahtbot links now :-/
<fanquake>
_aj: sounds like a situation we need to improve
Guyver2 has quit [Quit: Going offline, see ya! (www.adiirc.com)]
jespada has joined #bitcoin-core-dev
Cory has quit [Quit: Client closed]
Cory has joined #bitcoin-core-dev
l0rinc has joined #bitcoin-core-dev
<l0rinc>
#proposedmeetingtopic [IBD] multi-byte block obfuscation (part of #32043)
<l0rinc>
With the AutoFile batching now merged (#31551), seeking reviewers/reproducers for the block/index obfuscation optimizations in #31144 - achieving >20x performance gain and ~4% overall IBD speedup
peter_wrighten has quit [Ping timeout: 252 seconds]
adil has joined #bitcoin-core-dev
Christoph_ has joined #bitcoin-core-dev
adil has quit [Quit: adil]
Guest14 has joined #bitcoin-core-dev
Guest14 has quit [Client Quit]
kevkevin has joined #bitcoin-core-dev
kevkevin has quit [Ping timeout: 260 seconds]
l0rinc has quit [Quit: l0rinc]
Guyver2 has joined #bitcoin-core-dev
kevkevin has joined #bitcoin-core-dev
<vasild>
achow101: (or whoever hosts the meeting) I will not be at the meeting. The point of the topic I proposed is to get an estimate of how many people would be affected if we remove something that they use (GCC coverage from CMake as it is now)
janb84 has quit [Ping timeout: 244 seconds]
janb84 has joined #bitcoin-core-dev
sliv3r__- has quit [Read error: Connection reset by peer]
sliv3r__ has joined #bitcoin-core-dev
ghost43 has quit [Remote host closed the connection]
SpellChecker has quit [Read error: Connection reset by peer]
SpellChecker has joined #bitcoin-core-dev
ghost43 has joined #bitcoin-core-dev
Guyver2 has left #bitcoin-core-dev [Closing Window]
VonNaturAustreVe has joined #bitcoin-core-dev
VonNaturAustreVe has joined #bitcoin-core-dev
kevkevin has quit [Remote host closed the connection]
kevkevin has joined #bitcoin-core-dev
SpellChecker has quit [Remote host closed the connection]
SpellChecker has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] hodlinator opened pull request #32736: wallet: Warn upon failing to scan directory (master...2025/06/wallet_dir_iter) https://github.com/bitcoin/bitcoin/pull/32736
jespada has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
jespada has joined #bitcoin-core-dev
purpleKarrot has joined #bitcoin-core-dev
<lightlike>
vasild: I usually use gcc and have used lcov with it on occasion before, but clang is necessary anyway (sanitizers, fuzzing), so I don't really care, just need something that works and is documented.
<dergoegge>
More net_processing coverage than the in-repo fuzz tests, in all coverage metrics (line, function, region, branch) coverage (e.g. ~70% vs ~ 64% line coverage). Also coverage for reorgs went from nothing to something (i.e. Chainstate::DisconnectTip, Chainstate::MaybeUpdateMempoolForReorg)
dermoth_ is now known as dermoth
<dergoegge>
I’m still working on extending the fuzzer to increase coverage (e.g. no compact block support atm) and brainstorming what kind of bug oracles are feasible (please lmk if you have any ideas, join the fuzzing wg)
<purpleKarrot>
hi
<dergoegge>
Lastly, it would be interesting to have a collection of real p2p/validation bugs to test the fuzzer against (beyond what’s list ed on https://bitcoincore.org/en/security-advisories/). If you know of or have found bugs in your own or other PRs please let me know! Any kind of bug (logic, crash, …) is fine.
Christoph_ has joined #bitcoin-core-dev
rkrux has joined #bitcoin-core-dev
<marcofleon>
hype
<instagibbs>
nice re:reorgs
<rkrux>
hi
<eugenesiegel>
awesome about reorgs
<brunoerg_>
nice
<sipa>
hi
<sipa>
dergoegge: cool, will think about bugs that can be included
<achow101>
#topic Kernel WG Update (TheCharlatan)
<TheCharlatan>
I pushed a new version of #32427 including a write ahead log and crc32c checksums for each entry. Performance still seems to be a bit better compared to leveldb. This should make writes atomic and provide detection for potential corruption.
<corebot`>
https://github.com/bitcoin/bitcoin/issues/32427 | (RFC) kernel: Replace leveldb-based BlockTreeDB with flat-file based store by TheCharlatan · Pull Request #32427 · bitcoin/bitcoin · GitHub
<sipa>
TheCharlatan: (sorry for being lazy and not looking at code), what path structure are you using?
<cfields>
32427 spawned a discussion about a new on-disk block layout that is a bit of a distraction...
<sr_gi[m]1>
hi
<abubakarsadiq>
hi
enochazariah97 has joined #bitcoin-core-dev
<sipa>
cfields: jinx, ok...
<cfields>
I think it might be worth having that discussion here a bit..
<TheCharlatan>
path structures?
<cfields>
sipa: hah
<sipa>
TheCharlatan: directory/file names
hernanmarino has quit [Ping timeout: 268 seconds]
<l0rinc>
TheCharlatan: "Performance still seems to be a bit better compared to leveldb" - I have measured a reindex until 888888 blocks for your PR and the performance seemed roughly the same
enochazariah has quit [Ping timeout: 272 seconds]
<TheCharlatan>
oh, yeah, the PR does not touch the actual block storage hierarchy.
<sipa>
ah, ok
<cfields>
tldr: in the review for 32427, several reviewers brought up the idea of changing the on-disk block format while we're at it. One suggested example is one-file-per-block...
<dergoegge>
the on disk block index storage performance should not affect ibd at all, we keep all of it in memory no?
<sipa>
dergoegge: flushes to utxo set trigger flushes of block data, which could take some marginal amount of time?
<TheCharlatan>
I also think that given that we have some flexibility with the existing pair of 64bit numbers to point to a location on disk, we might be able to use the encoding for single block files.
<cfields>
for 32427 to move forward, we either need to agree to punt on that discussion and bring it up again later, or bikeshed some layout changes. For the sake of not grinding the flat index changes to a halt, I prefer the former.
<dergoegge>
sipa: ah right, ty
<vasild>
"one-file-per-block" ... some filesystems may not handle well 1M files, is there any benefit from doing that?
<l0rinc>
won't we have problems with so many open files on Windows - like we had with LevelDB?
<sipa>
vasild: they can be grouped into subdirectories too, either by hash, or (my suggestion, i think) by height
<cfields>
vasild: you don't really need an index anymore if the blocks can be located using the filesystem alone.
<vasild>
sure, I do not mean 1M files in a single directory
<cfields>
er, a much simpler index at least.
<sipa>
yes, agree with that, don't put 1M files in a single directory :)
<l0rinc>
would be really cool to try it out though - let me know if I can help with benchmarking anything
<TheCharlatan>
dergoegge yes, in ibd time it is just good to know there are no regressions. The speedup during block index loading when starting a node is more significant though.
<lightlike>
people tend to copy datadirs for backups etc. years ago that would take very long with millions of files, did that change?
<sipa>
cfields: no need for the block data structure change (if any) to be made in the same PR, it suffices if it's in the same release
enochazariah97 has quit [Quit: Client closed]
freedomcode has quit [Remote host closed the connection]
reardencode has joined #bitcoin-core-dev
<sipa>
cfields: well "index" isn't the right word anymore then, but we still have a block database with things like validation status and so on (though, arguably, that belongs more in validation then block storage)
<vasild>
cfields: that would be like outsouring the indexing to the OS/filesystem, e.g. blocks/1/15/153/153764 (for block 153764). The question is can we do better than the worst OS/FS that is going to use Bitcoin Core, or even, better than the best FS?
<l0rinc>
would that mean that leveldb indexes can point to files and indexes now instead of duplicating any data? Or is that still problematic for pruned nodes?
<sipa>
it'd be pretty awesome if the block index can go away, and its remaining fields moved into the chainstate
<vasild>
I guess it has to be tested...
<cfields>
sipa: right
<sipa>
though maybe not more than an aesthetical advantage
<sipa>
it's just ugly that we store validity of a block in the block index, it's not a storage-related thing but a validation-related one
<cfields>
sipa: much easier for non-core software to read blocks. definitely not just aesthetic.
<sipa>
cfields: i'm talking about the block index going away, independent of block storage structure used
<sipa>
lightlike: yeah, i worry about that too - maybe in these days with fast SSDs that's less of an issue, but i don't actually know
<cfields>
sipa: not sure what you mean. how does it go away if we still need to map them to a non-addressable location?
<TheCharlatan>
also seems useful though to have block validity included for as long as we can't read the chainstate easily from external sources.
<sipa>
cfields: sorry, i'm being confusing. I'm talking about the advantage of the block index going away, which is something that's only possible with a change to directory structure - but i'm just talking that that index going away itself is nice, ignoring the advantage that the directory structure may bring without that
<sipa>
TheCharlatan: i disagree!
<sipa>
TheCharlatan: it's pretty counterintuitive that someone just copying the block data, but not chainstate, would inherit the validity status from that block data
<cfields>
sipa: ah, yes, completely agree.
<TheCharlatan>
mmh, so you couldn't recreate the block tree anymore without re-validating?
<sipa>
TheCharlatan: i just wouldn't think of the validity status as part of the block data
<sipa>
you can recreate it from just disk, but without a chainstate to fill in validity status, every block would be marked as non-validated
Christoph_ has joined #bitcoin-core-dev
<sipa>
this is probably getting a bit into the weeds for this meeting
<TheCharlatan>
yes :)
<lightlike>
i feel like the discussion of this idea deserves a separate issue, it's not that related to ThCharlatan's PR.
<TheCharlatan>
sooo... next topic?
<achow101>
next topic
<achow101>
#topic Benchmarking WG Update (josie, l0rinc)
<l0rinc>
I have opened the obfuscation PR in Oct 2024, split off other smaller optimizations since, some of those are merged already - the XOR block obfuscation one is next, reproducers and reviewers are welcome
<l0rinc>
I've received 2 raspberry pi servers recently, I will be able to do benchmarks on those as well (already measured Sipa's #32545) - since many people are complaining they're IBDs are taking weeks
<jonatack>
maybe you are already doing so, but benchmarking with slow internet connections for ibd/p2p might reveal additional issues in how a change under consideration impacts nodes under more hostile conditions like that
Cory has quit [Quit: Client closed]
<l0rinc>
the PI server are also on slightly slower network and lower memory (4 & 8GB)
Cory has joined #bitcoin-core-dev
<dzxzg>
it's challenging to create realistic bad network conditions in a way that is also reproducible
<l0rinc>
if you think it's important, I can simulate slow connections as well to see what that reveals (the Hetzner servers I have are very fast for normal IBD as well, I guess the nodes it find are probably also from Hetnzer)
<jonatack>
benchmarking how bitcoind holds up under tough conditions seems valuable (i find it does very well currently, apart from ibd time)
<jonatack>
(and the stalling/timeout->disconnect logic)
<sipa>
yeah i think those are very different things to measure
<l0rinc>
I've also checked swiftsync with very simpmple xor instead of muhash - and locally on my laptop it finished a reindex until 888888 blocks in 29 minutes
<sipa>
pure computation/disk/io based effects
<abubakarsadiq>
not sure whether it will help, but having a node with few peers as well? like doing ibd without inbounds, someone complained to me that their ibd was slow.
<abubakarsadiq>
when I checked it due to having very few peers, basically no inbounds at all.
<sipa>
vs network block IBD fetching effects
<eugenesiegel>
I have heard that netem can be used to simulate latency
<achow101>
sorry, going to try to keep things moving since we have 2 actual topics this week
<achow101>
we can have further discussion after the meeting, or in an issue on github
<l0rinc>
k, thanks
<achow101>
#topic Cluster Mempool WG Update (sdaftuar, sipa)
<sipa>
not much to report
<sipa>
next to review remains reorg support in txgraph, #31553
<johnny9dev>
Significant progress was made this week on rebasing the project and moving it to CMake+Qt6. Both hebasto and pinheadmz have PRs now to test and fix issues with the update. bitcoin-core/gui-qml#470 bitcoin-core/gui-qml#472. Both PRs are now able to load up the QML app and get to the main navigation and views.
<johnny9dev>
For myself, I've been focused on the current Send and Recieve PRs that are in review. I've also been making updates to how address inputs are handled, bitcoin amount label formatting, and address label formatting to get closer to the design spec.
<achow101>
#topic Script Validation WG Update (fjahr)
<fjahr>
Nothing really noteworthy this week, more time for the *actual* topics ;)
<achow101>
#topic Erlay WG Update (sr_gi, gleb)
<sr_gi[m]1>
Nothing much to report on my end either, I've been off a couple of weeks. Currently picking warnet sims where I left them before leaving. Will report once I'm finished
<achow101>
#topic Rough poll - how many people use GCC and how many of those do generate code coverage with GCC? (vasild)
<vasild>
Trying to get an estimate of how many people are using the current GCC-oriented coverage we have in CMake.
<eugenesiegel>
I use gcc for code coverage
<achow101>
I use gcc, but I haven't generated coverage with it in ~5 years (or any other coverage really)
<sipa>
i use GCC for creating benchmarking builds, and clang for fuzzing builds; i haven't used coverage tools for a long time (maybe i should?)
Guest4319 has joined #bitcoin-core-dev
<fjahr>
Only have been using clang for anything cov related since that was what was recommended to me, but definitely not a heavy user lately
<eugenesiegel>
coverage reports are nice for telling you where the fuzzer is stuck
<fanquake>
eugenesiegel: are you using GCC for fuzzing? Or just switching to it for coverage?
<eugenesiegel>
fanquake: I use libfuzzer to fuzz on a VM, and then run the corpus through with gcc coverage on my mac
<fanquake>
gcc on a mac is clang though? So brew installed GCC?
<l0rinc>
I usually use test code coverage in cLion which automatically adds the `-DCMAKE_CXX_FLAGS="-fcoverage-mapping -fprofile-instr-generate" -DCMAKE_C_FLAGS="-fcoverage-mapping -fprofile-instr-generate"` options
<dergoegge>
l0rinc: and then you (or cLion) uses the llvm tools to create the actual report?
<l0rinc>
there's a built-in report, haven't checked how it generates it
<eugenesiegel>
fanquake: oh, I didn't know that. I use gcov so I assumed it was gcc
<darosior>
vasild: most of the time i use gcc for default builds and clang for fuzzing or using sanitizers. I generate source-based coverage with LLVM. I have never used gcov based coverage as presented in our doc.
Christoph_ has quit [Quit: Christoph_]
Guest4319 has quit [Remote host closed the connection]
eugenesiegel has quit [Quit: Client closed]
eugenesiegel has joined #bitcoin-core-dev
cotsuka has quit [Remote host closed the connection]
cotsuka has joined #bitcoin-core-dev
eugenesiegel has quit [Quit: Client closed]
l0rinc has joined #bitcoin-core-dev
Talkless has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 265 seconds]
<vasild>
darosior: looks like one more confirmation of "< dergoegge> so no one is using gcc is the conclusion?" :)
l0rinc has quit [Quit: l0rinc]
Cory93 has joined #bitcoin-core-dev
jonatack has joined #bitcoin-core-dev
Cory has quit [Ping timeout: 272 seconds]
<hodlinator>
vasild: I've used GCC mainly for getting benchmarks resembling release builds, otherwise I'm happy to use Clang. Haven't really used coverage that much yet.
purpleKarrot has quit [Quit: purpleKarrot]
<fanquake>
CoreCheck is using it, however speaking to Max, he was already looking into migrating to LLVM/Clang
VonNaturAustreVe has quit [Ping timeout: 260 seconds]
purpleKarrot has joined #bitcoin-core-dev
Cory15 has joined #bitcoin-core-dev
Cory93 has quit [Ping timeout: 272 seconds]
Talkless has quit [Quit: Konversation terminated!]
dyscope has joined #bitcoin-core-dev
dyscope_ has joined #bitcoin-core-dev
dyscope_ has quit [Remote host closed the connection]
dyscope has quit [Remote host closed the connection]
dyscope has joined #bitcoin-core-dev
dyscope has quit [Remote host closed the connection]
dyscope has joined #bitcoin-core-dev
VonNaturAustreVe has joined #bitcoin-core-dev
VonNaturAustreVe has quit [Ping timeout: 248 seconds]
robobub has quit [Quit: Connection closed for inactivity]
bugs_ has quit [Quit: Leaving]
purpleKarrot has quit [Quit: purpleKarrot]
<bitcoin-git>
[bitcoin] danielabrozzoni opened pull request #32740: refactor: Optimizations & simplifications following #25717 (master...upforgrabs/25968) https://github.com/bitcoin/bitcoin/pull/32740
dyscope has quit [Quit: Leaving]
Guest22 has joined #bitcoin-core-dev
Guest22 has quit [Client Quit]
jespada has quit [Ping timeout: 260 seconds]
jespada has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 248 seconds]
kevkevin has quit [Remote host closed the connection]
<bitcoin-git>
[bitcoin] waketraindev opened pull request #32741: rpc: add optional nodeid param to filter getpeerinfo (master...2025-06-getpeerinfo-filterid) https://github.com/bitcoin/bitcoin/pull/32741
conman has quit [Remote host closed the connection]
kevkevin has joined #bitcoin-core-dev
conman has joined #bitcoin-core-dev
jonatack has joined #bitcoin-core-dev
kevkevin has quit [Remote host closed the connection]
<bitcoin-git>
bitcoin/master 0a24870 Ryan Ofsky: indexes, refactor: Stop requiring CBlockIndex type to call IsBIP30Unspenda...
<bitcoin-git>
bitcoin/master 6f1392c Ryan Ofsky: indexes, refactor: Remove remaining CBlockIndex* uses in index Rewind meth...
<bitcoin-git>
[bitcoin] achow101 merged pull request #32694: index: move disk read lookups to base class (master...2025_indexes_remove_CBlockIndex_access) https://github.com/bitcoin/bitcoin/pull/32694