Guyver2 has left #bitcoin-core-dev [Closing Window]
adil has joined #bitcoin-core-dev
adil has quit [Quit: adil]
<l0rinc>
sipa,andrewtoth: please see my measurements in https://github.com/bitcoin/bitcoin/pull/31132#issuecomment-3646563048 where I've compared dbcache 100-7000. There is some difference between dbcache=400 (03:56:25 for 900k blocks) and dbcache=2000 (03:33:51 for the same), but after that there isn't any measurable speedup - unless it's on network storage or hdd.
<bitcoin-git>
[bitcoin] fanquake opened pull request #34623: Update secp256k1 subtree to latest master (master...secp256k1_subtree_update) https://github.com/bitcoin/bitcoin/pull/34623
andrewtoth_ has joined #bitcoin-core-dev
andrewtoth has quit [Remote host closed the connection]
timbo_xyz has quit [Quit: WeeChat 4.8.1]
apollodorus has joined #bitcoin-core-dev
brunoerg has joined #bitcoin-core-dev
brunoerg_ has joined #bitcoin-core-dev
brunoer__ has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 252 seconds]
brunoerg_ has quit [Ping timeout: 246 seconds]
<bitcoin-git>
[bitcoin] HowHsu opened pull request #34625: cluster_linearize: add tests and benchmarks for chain and tree-shaped clusters (master...linearize-tests) https://github.com/bitcoin/bitcoin/pull/34625
brunoer__ has quit [Remote host closed the connection]
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 245 seconds]
eugenesiegel has joined #bitcoin-core-dev
brunoerg has joined #bitcoin-core-dev
jurraca has joined #bitcoin-core-dev
memset has quit [Remote host closed the connection]
memset has joined #bitcoin-core-dev
eugenesiegel has quit [Ping timeout: 272 seconds]
memset has quit [Remote host closed the connection]
memset has joined #bitcoin-core-dev
Novo has joined #bitcoin-core-dev
sr_gi has joined #bitcoin-core-dev
eugenesiegel has joined #bitcoin-core-dev
dzxzg has joined #bitcoin-core-dev
Emc99 has joined #bitcoin-core-dev
maxedw_ is now known as maxedw
<fjahr>
#startmeeting
<corebot>
fjahr: Meeting started at 2026-02-19T16:00+0000
<Novo>
Works pretty well, but might require changes to the API
<Novo>
In practice, users should not need to do this since worst case scanning time with a maximum of 1000 outputs is in seconds
<Novo>
Hence, there's not a lot of support to modify the API to support parallel scanning but we will see
<theStack>
for the proposed K_max protocol change, I'm planning to open a BIP-352 PR today or tomorrow, with proper test vectors; no one has opposed the K_max=1000 suggestion, but different suggestions could still be made in that PR
<Novo>
Nice theStack
<Novo>
Nothing else to add
<fjahr>
#topic Cluster Mempool WG Update (sdaftuar, sipa)
<sipa>
We're getting very close with cluster mempool. #34023 was merged, and my last remaining PR is #34616 which I'd like to see in. There are a few more follow-ups like extra tests, RPC output, optimizations, and possible interaction with mining (to decide when to wake a waiting miners / cache blocks), but I think those aren't essential for feature freeze.
<fjahr>
andrewtoth: Thanks, I was just looking for it :)
<fjahr>
It's still a pretty long list, I didn't have time to go through it yet. Any comments?
<fjahr>
Otherwise, please review!
<fanquake>
I don't think there's anything that would require pushing back. Things just need review. Plenty of double-ups where it's the bug report + PR both tagged.
<fanquake>
Mostly bug reports for bugs introduced in this cycle.
ghost43 has quit [Remote host closed the connection]
<sipa>
Hi, I've been thinking about this for a while, and it seems to me that the default dbcache setting (450 MiB) is a bit outdated for most systems these days.
<l0rinc>
let me repeat what I posted here a few hours ago: `sipa,andrewtoth: please see my measurements in https://github.com/bitcoin/bitcoin/pull/31132#issuecomment-3646563048 where I've compared dbcache 100-7000. There is some difference between dbcache=400 (03:56:25 for 900k blocks) and dbcache=2000 (03:33:51 for the same), but after that there isn't any measurable speedup - unless it's on network storage or hdd.`
<l0rinc>
`But note that after the PR validation with 100 mb dbcache is still slightly faster than than master with 7 GB. Extracted a few other measurements from the PR to`
Guest80 has joined #bitcoin-core-dev
<darosior>
It has been outdated, but with the current RAM prices, it's back to totally fine :p
<sipa>
darosior: haha
<theStack>
:D
<l0rinc>
as andrewtoth mentioned yesterday #31132 modifies this assumption and even dbcache=100 will be faster than master
<sipa>
I understand that there are users who still run on minimal hardware, for which a default bump might not be appropriate, but still - I don't think there is really any problem with e.g. if you're on a 64-bit system and have say over 8 or 16 GiB of RAM, defaulting to 2 GiB dbcache
Guest80 has quit [Client Quit]
<darosior>
1GiB should be totally fine in my opinion, but i defer my judgement to people that actively monitor that area (like Lorinc and Andrew who chimed in)
epicleafies has joined #bitcoin-core-dev
<l0rinc>
I can barely run an IBD on a node with 2 GB memory
<andrewtoth_>
Yeah I think it would be a big speedup for network connected storage, which I think is the majority of nodes being run
<l0rinc>
and after Andrew's parallelization change the difference won't be that big - can we postpone this decision after it lands?
<andrewtoth_>
I'm more concerned about steady state than IBD. We don't periodically clear the cache anymore unless it grows too big. So, today users can run it for months/years and it won't exceed 450MB. If we update this default then maybe users who run their nodes on low memory systems and just set and forget will come back to an OOM crashed node after some months.
<darosior>
Hasn't the expectation always been that people running on minimal hardware tweak their settings anyways? We have docs for this, and specs for minimal hardware must have been a lot lower a decade ago when we had the same default.
<sipa>
l0rinc: my (largely unsubstantiated) guess is that there are settings (super slow spinning USB-connected disk) with very slow I/O, where even with parallel fetch, a large dbcache makes a big difference
Murch[m] has quit [Changing host]
Murch[m] has joined #bitcoin-core-dev
<darosior>
2 GiB sounds good to me as a default too, for what it's worth.
<fjahr>
It is also notable that this may be shared with the indexes if they are active
<Murch[m]>
It’s my impression that especially a lot of the low-powered devices are run by people that don’t configure…
<l0rinc>
yes, as mentioned above, HDDs are definitely affected even after the parallel fetcher
<l0rinc>
most people I talked to (including node providers) are afraid to change the defaults
afiore has quit [Remote host closed the connection]
<pinheadmz>
Murch[m] i wouldnt count on that, arent there plenty of SE questions about running on a Pi? suggested confs are easy to find
afiore has joined #bitcoin-core-dev
<darosior>
Maybe a more conservative bump to 1GiB then?
<dzxzg>
It would be nice if out-of-memory behavior was better before/along with raising the default, e.g. monitoring memory pressure / available memory and dynamically decreasing sizes of caches to avoid OOM
<_aj_>
l0rinc's graph suggested 1GB would get most of the benefit, i thought
<l0rinc>
yes, we have a warning now if the dbcache is too high compared to the total ram
<l0rinc>
I could get behind that if we separated dbcache during and after IBD (which we partially do already by stealing the mempool memory)
<sipa>
fwiw, i'm motivated by seeing people complain about days-long syncs - which usually is a sign there is more wrong than just dbcache, but it keeps reminding me that i think 450 MiB is silly today
<Murch[m]>
Well, it is plenty in the steady state at the chaintip…
ghost43 has quit [Remote host closed the connection]
<sipa>
yeah, this is just about IBD
<hodlinator>
couldn't it be a sign of trying to fit the UTXO set in the -dbcache and running into swapping issues?
ghost43 has joined #bitcoin-core-dev
<hodlinator>
...due to running out of RAM
<sipa>
hodlinator: not in the recent example (#34601) i think, but yes
<corebot>
https://github.com/bitcoin/bitcoin/issues/34601 | bitcoind sync gets very, very slow on Augst 2023 in btc chain (maybe 2/3 progressed or something like that) · Issue #34601 · bitcoin/bitcoin · GitHub
<Murch[m]>
Yeah, I heard that malarkey a few times, too, that people thought dbcache has to be bigger than the UTXO set. But we should catch that with the warning, right?
<l0rinc>
it could, that's why we have the new warning now - and why I opened #34606
<andrewtoth_>
l0rinc: agreed, switching to lower dbcache after IBD would make this safer
<hodlinator>
yes, it should warn, provided the user runs a newer version.
<andrewtoth_>
bumping to 2GB during IBD on 64 bit systems then lowering to current default after IBD would be great
rkrux has quit [Quit: Client closed]
<jonatack>
hi
<darosior>
andrewtoth_: why safer? More optimal maybe, but i don't see how a larger dbcache makes things less safe at tip.
<_aj_>
good grief, the utxo set via usb2?
<sipa>
andrewtoth_: that seems weird to me, if you have enough memory for one, why not use it all the time?
<l0rinc>
I don't mind pushing a PR that creates a 2 gb default for anything having e.g. 8 gb total memory and reverting to 450 after IBD
<sipa>
if the concern is OOMing later, we can try a preallocation run - just fill the dbcache once at startup and wipe it
<l0rinc>
but even 2 GB is too high for a 4 GB machine
<Murch[m]>
sipa: People run all sorts of other stuff on their Bitcoin boxes now, but most of it only after IBD. So they might have more RAM for bitcoind before IBD is finished
<sipa>
yeah fair
<andrewtoth_>
what Murch said
<darosior>
I see
<darosior>
Would everyone be happy with a 1GiB default? That's already more than twice as better as today, and could mitigate the concerns raised here?
<dzxzg>
It would be good to know connection times at the tip as a function of dbcache size
<l0rinc>
I have regular OOMs on 2GB and 4GB boxes ... everything seems fine and after a while something triggers an OOM even with the safety nets. These are so close to the limit by default
<darosior>
s/twice as better/twice better/ oui oui baguette
<andrewtoth_>
dzxzg: I am trying to measure that for #31132. With default dbcache today the cache is not flushed for about 3 weeks. So it's a long thing to measure.
<Murch[m]>
1 GB seems like an improvement. It would be great if we could do something conditionally on how much RAM the system has
<darosior>
l0rinc: by this token we will never be able to bump defaults ever again, because someone will still have a 4GiB box somewhere a decade from now
<darosior>
We have configuration for a reason, defaults should represent the vast majority of users, not the lowest common denominator possible?
<fjahr>
1 GB sounds like a good compromise to me, again, also keep in mind this may be shared with the indexes
<lightlike>
is the suggested bump meant to go into v31 still?
<sipa>
i also like the idea of having presets, like are you running in a very resource-constrained system, or a giant machine? which could control things like dbcache, mempool size, network buffers, connection counts, ...
<_aj_>
pruning
<sipa>
which may be more interesting for config-averse people to choose than manually tweaking values
<Murch[m]>
sipa: That sounds interesting.
<darosior>
sipa: makes sense.
<l0rinc>
I'm all for e/acc, it's just something I see on the lower-end devices - I would go for 100 if lower than 2 GB, 400 if lower than 4 GB and 1 GB if higher
<andrewtoth_>
l0rinc: I think you need to configure to get it to run on 2GB at all, right?
<_aj_>
would just providing example three configs be a good/easy way of doing that?
<l0rinc>
yes, that already needs a -dbcache=100
<dzxzg>
I think it's confusing to have to tweak the size of each of bitcoind's memory pools individually, instead of setting a single resource usage target
<Murch[m]>
Maybe we can reconsider that in light of it’s 10th anniversary coming up ^^
<sipa>
(its)
<andrewtoth_>
I think 1GB would be an improvement. Put it front and center in the release notes.
<darosior>
yay
<sipa>
100 / 400 / 1000 depending on ram size makes sense to me
<Murch[m]>
In before “Bitcoin Core hates Raspberry Pi users!”
<Murch[m]>
:p
<andrewtoth_>
lol
<sipa>
yes, i mean for 31.0
<andrewtoth_>
sipa, l0rinc: yes, 100/400/1000
<Murch[m]>
sipa: I thought that it was not available on all systems or smth
<l0rinc>
"Put it front and center in the release notes" - clarifying that this isn't because we want more spam :p
<Murch[m]>
As in, we don’t know in Bitcoin Core how much the system has?
<sipa>
Murch[m]: we always know if we're on 32-bit vs 64-bit, unsure about accessing ram size on every OS
<jonatack>
presets changing multiple config options as a user-friendly suggestion sounds good indeed
<_aj_>
it's only going to be linux/bsd that have low memory though?
<sipa>
_aj_: i'd guess that many people run on old windows boxes
<jonatack>
if feasible, avoiding too much devilry in the details
<Murch[m]>
well then, default based on available RAM sounds like a great idea!
<sipa>
ok i think we can conclude this topic and discuss on a PR if someone(?) creates one
<darosior>
Murch[m]: doesn't that contradict your previous point that it's going to make the process OOM if the user runs anything else besides bitcoind?,,
<l0rinc>
I'll push a PR for that today, thanks for the feedback. To be clear, should we revert to a lower value after IBD or should we choose values that make sense regardless (excluding the barrowed mempool 300 MB)?
<fjahr>
I didn’t have time to prepare anything for this but since it came up and might be relevant for the wider group, I thought I would make everyone aware of the ideas in the comment thread there.
<fjahr>
tldr; the ideas are: moving the asmap-data repo into bitcoin-core org, letting participants of the collaborative runs sign their results and transcript with the necessary script and place for sharing, sharing the raw data before processing, notifying users of an older asmap data (potentially only after default).
<darosior>
l0rinc: in my opinion this should be a separate improvement.
<fjahr>
Anything that people would like to add or comment on there? I will collect and track them in the asmap issue.
<sipa>
moving asmap-data to bitcoin-core org makes sense, as does having some kind of attestation process for collaborative asmap creation participants, effectively signing off on "i ran kartograf with these settings at this point in time, and obtained this"
stringintech has quit [Remote host closed the connection]
<Murch[m]>
darosior: That was meant to be a point why 1 GB might be too big, when a box has only 4 GB
<Murch[m]>
But if we generally pick defaults based on RAM, we’d probably do something smaller for a 4 GB box?
Emc37 has joined #bitcoin-core-dev
<l0rinc>
I'll add a PR for that and we can discuss there, thanks for the feedback!
<Murch[m]>
Sorry, can talk more later. Different topic now.
<fjahr>
Ok, I guess the RAM question is still more exciting than this question. Still feel free to comment here or in the links later if you have opinion.
<fjahr>
Anything else to discuss?
<andrewtoth_>
fjahr congrats on getting asmap merged!
<darosior>
Are people interested in picking back up the GUI topic from last week. I didn't attend but reading the backlog it doesn't seem like much was decided.
<Murch[m]>
I haven’t looked much into it, but if we’re shipping the ASMAP default, putting the attestation into the Bitcoin Core org makes sense
<fjahr>
andrewtoth: thank you :)
<darosior>
I guess i didn't see hebasto nor achow and we'd need them to discuss GUI stuff..
<hebasto>
darosior: hi
<fjahr>
Murch: default on is for the future but the process should keep that in mind already I think
<darosior>
hebasto: anything to add, following up on last week's discussion? What should be the way forward?
Emc99 has quit [Ping timeout: 272 seconds]
bugs_ has joined #bitcoin-core-dev
<fjahr>
I think the outcome was we should wait with a final decision until we meet in person?
<fjahr>
Seems like people are exploring options, like will's vibe code rpc experiment
<darosior>
It does not seem like any of the discussion we've had in person have ever made this move forward.
<fanquake>
It'd be good to know about #33684 (given the Qs there, or maybe have a dedicated issue). There doesn't seem to be an answer to where new features should be added? Seems like something we should know, to unblock progress/prevent wasted work/have any clarity.
<johnny9dev>
I did some work as well since last week to get everything for gui-qml in a PR or merged so all work from last year is in the default branch for it. I have also given hebasto a feature comparison summary
<hebasto>
^ this
<fjahr>
darosior: is that a problem with the topic or meeting in person?
<darosior>
:)
<fanquake>
johnny9dev: so you're working on something that is going to whotely replace the current widgets gui right?
<fanquake>
*wholely
<johnny9dev>
I think so.
<sipa>
wholly?
<darosior>
So is the plan to replace the QT widget GUI with the QML GUI?
<darosior>
Would this be IPC based?
<hebasto>
not now
<fjahr>
I think having some time to think about it and explore options is good, I haven't had time to look into it more because of getting stuff done for feature freeze
<johnny9dev>
last year I wouldnt have agreed to such a think but today with the tools we have. i might
<fanquake>
johnny9dev: great, can we get a writeup or similar, that explains the architecture, dependencies needed, how are we going to decouple it from the current code, etc
<sedited>
darosior's delving post was also discussed here, which at least gives a taste of what gui users might think of it https://stacker.news/items/1436922
<fjahr>
Maybe a way to encourage this would be to let people note down where they currently stand in a table similar to what we had on controversial issues in the past? Does that sound interesting?
<johnny9dev>
yeah, I will work towards that and get some help from hebasto as well
<fanquake>
I guess this means that currently, nothing is being added to the gui, until qml ships?
<hebasto>
correct
<hebasto>
once the release process is safe regarding more gui deps, we will be able to switch to qml
<darosior>
sedited: oh, i hadn't seen that, thanks.
<fanquake>
It's not just the release process. i.e the whole subtree thing needs to be fixed/
<dzxzg>
given that the timeline for qml is not bounded, does it make sense to stop adding features to qtwidgets?
<hebasto>
subtree is minor issue, I have a wip branch
<fanquake>
and process separation?
<hebasto>
dzxzg: it is de-facto
<hebasto>
process separation is a next step
<dzxzg>
hebasto: To make sure I understand, you mean that as a fact, no one is adding features to qtwidgets today?
<fanquake>
I take it nothing is blocked on designers now either?
<hebasto>
right
<Murch[m]>
interesting thread, sedited
<hebasto>
dzxzg: noone is actively reviewing already suggested features
<darosior>
What's the status of the QML GUI, as in, in your branch johnny9dev? My understanding was that it was still under heavy development, but is it now close to feature complete?
<hebasto>
feature comparison tables will be available shortly
<dzxzg>
hebasto: That makes sense, I would just oppose it as a matter of policy, e.g. if people do start showing up and reviewing gui PR's, I don't think they should be rejected on the basis of the qtwidgets gui being replaced with qml,
<hebasto>
dzxzg: sure
nervana21 has joined #bitcoin-core-dev
<darosior>
dzxzg: Is adding new features to the GUI even a goal? My understanding was that its main purpose was backward compatibility to support existing users, but that if you wanted a neat GUI with all the new features, other projects already do it better, and are a more appropriate place for further development.
<johnny9dev>
there is work to do for sure. working towards understanding how close at the moment.
<fjahr>
3 minutes left...
<sipa>
darosior: people might not agree on what the goal is
<dzxzg>
darosior: I mean it may not be a goal for you
<pinheadmz>
darosior personally, if the gui had harwarde wallet support i would never need another bitcoin program, i want that for myself ;-)
<sipa>
we could agree on explicit goals, but that doesn't align well with the notion of not telling people what they should work on
<darosior>
it's not like it does not come with its own externalities
<pinheadmz>
so i guess that feature should go in QML then ?
<fanquake>
sipa: ideally we can agree on goals like, not wasting developers time, by not having any clarity around what the project is doing
<fanquake>
i.e: someone shows up implementing a new gui feature, unaware that nothing is going to be added to the current gui, or they should have added it to qml
<fjahr>
johnny9dev: If we continue with the qml gui as originally planned I think it would be good to make a plan for attracting new contributors in a more serious manner as well
<sipa>
fanquake: agreed, but i think it's more a matter of setting expectations than strictly deciding what is being worked on
<darosior>
If it was a separate program in a separate project, i agree, if people are interested in doing stuff, good for them, good for their users. But the reality is not so.
<fanquake>
given that qml has taken 5 years, and not shipped, seems like a good time to try clarify some of these things
<dzxzg>
it kind of seems like shooting oneself in the foot to stop all qtwidgets gui development in the hope that someday it's replaced by qml, if you dislike the gui you may not see it that way since it perpetuates the present problems with the gui
<Murch[m]>
It’s always right before something is done that people get disenchanted? :p
<fanquake>
hebasto: I hope qml wasn't waiting 5 years for me to leave that comment
eugenesiegel has quit [Quit: Client closed]
<sipa>
my point is mostly a response to dzxzg... if some group of people were hypothetically to show up and start working on, and reviewing and testing qtwidgets stuff, then i don't see the problem with those being merged... but getting clarity on whether such a group or not exists matters, and may inform people to not take on such projects if there isn't
<fanquake>
(I think splitting up the release builds is also tangentially, given it would already be useful, regardless of wether we are switching gui backends)
Emc37 has quit [Quit: Client closed]
nervana21 has quit [Quit: Client closed]
nervana21 has joined #bitcoin-core-dev
Bob2026 has quit [Quit: Client closed]
<sipa>
(lunchtime)
jurraca has quit [Quit: leaving]
SakshiKasat18 has joined #bitcoin-core-dev
<hebasto>
fanquake: yes, but it is always someone expresses an idea crystal clearly
sr_gi has quit [Quit: Client closed]
<bitcoin-git>
[bitcoin] fanquake opened pull request #34627: guix: use a temporary file over sponge, drop moreutils (master...replace_sponge) https://github.com/bitcoin/bitcoin/pull/34627
<bitcoin-git>
[bitcoin] sedited merged pull request #34385: subprocess: Fix `-Wunused-private-field` when building with clang-cl on Windows (master...260122-clangcl-unsused) https://github.com/bitcoin/bitcoin/pull/34385
epicleafies has quit [Quit: Client closed]
cfields has joined #bitcoin-core-dev
nervana21 has quit [Quit: Client closed]
<cfields>
grr, my ircd was acting up for today's meeting :(
<cfields>
will read the log
dzxzg has quit []
<johnny9dev>
fjahr: re: recruiting new devs. I actually might have a new dev that is interested in helping get the qml gui feature complete so working on onboarding him now. I'm hopeful about this aspect of the project
<jarolrod>
I will be helping with review on the qml side, and can help attract new contribs
eugenesiegel has joined #bitcoin-core-dev
<cfields>
johnny9dev: (I missed the meeting, catching up now) what interface does the qml interface use to communicate with Core?
<hebasto>
cfields: internal
<cfields>
is there a branch I can take a look at? I'd be interested in seeing what it would take to do it over IPC.