kevkevin has quit [Remote host closed the connection]
cmirror has quit [Remote host closed the connection]
cmirror has joined #bitcoin-core-dev
kevkevin has joined #bitcoin-core-dev
kevkevin has quit [Ping timeout: 252 seconds]
kevkevin has joined #bitcoin-core-dev
kevkevin has quit [Ping timeout: 252 seconds]
mcey has joined #bitcoin-core-dev
emcy__ has quit [Remote host closed the connection]
eragmus_ has joined #bitcoin-core-dev
kevkevin has joined #bitcoin-core-dev
kevkevin has quit [Ping timeout: 246 seconds]
adil has quit [Ping timeout: 252 seconds]
adil has joined #bitcoin-core-dev
kevkevin has joined #bitcoin-core-dev
kevkevin has quit [Ping timeout: 244 seconds]
flooded has quit [Ping timeout: 272 seconds]
<laanwj>
sipa: i checked. as i understand, multiprocess uses two packages: Libmultiprocess and LibmultiprocessNative (same for a non-cross build), you can set their root location with "-DLibmultiprocess_ROOT=..." and "-DLibmultiprocessNative_ROOT=..." where ... is the directory that contains `LibmultiprocessConfig.cmake`
<laanwj>
this should probably be documented somewhere, i can imagine it to be quite common that people install their own libmultiprocess, it's not a thing commonly installed
<bitcoin-git>
[bitcoin] l0rinc opened pull request #31144: optimization: pack util::Xor into 64 bit chunks instead of doing it byte-by-byte (master...l0rinc/optimize-xor) https://github.com/bitcoin/bitcoin/pull/31144
adil has joined #bitcoin-core-dev
kevkevin has quit [Ping timeout: 245 seconds]
kevkevin has joined #bitcoin-core-dev
Guyver2 has joined #bitcoin-core-dev
Guyver2 has left #bitcoin-core-dev [#bitcoin-core-dev]
<bitcoin-git>
bitcoin/master a0c9595 laanwj: doc: Make list of targets in depends README consistent
<bitcoin-git>
bitcoin/master 68f29b2 merge-script: Merge bitcoin/bitcoin#31141: doc: Make list of targets in depends README c...
<bitcoin-git>
[bitcoin] fanquake merged pull request #31141: doc: Make list of targets in depends README consistent (master...2024-10-depends-platforms) https://github.com/bitcoin/bitcoin/pull/31141
<achow101>
There is 1 pre-proposed meeting topics this week. Any last minute ones to add?
<lightlike>
Hi
<achow101>
#topic Working Groups (achow101)
<achow101>
Last week at CoreDev, we had a discussion about priority projects and how they don't seem to be working anymore. So we're going to try something a bit different - working groups.
<achow101>
The idea is that people who want to work on a project together will form a working group to focus on that particular project, and will then give updates in the weekly irc meeting
<laanwj>
hi
BlueMatt[m] has joined #bitcoin-core-dev
<BlueMatt[m]>
I’d like to discuss sv2 and the new bitcoin core nih mining protocol.
<sdaftuar>
i think that's it from me, i know sipa is working on the TxGraph code which will be important to review when he finishes
<sipa>
any week now
<achow101>
soon(tm)
<achow101>
#topic MuSig2 WG Update (achow101)
<achow101>
waiting for libsecp to do a release with the MuSig2 module
<sipa>
any week now
<achow101>
also working through a couple of rebase issues in #29675
<gribble>
https://github.com/bitcoin/bitcoin/issues/29675 | wallet: Be able to receive and spend inputs involving MuSig2 aggregate keys by achow101 · Pull Request #29675 · bitcoin/bitcoin · GitHub
<achow101>
will perhaps try to split it up to make it easier to review
<achow101>
#topic Legacy Wallet Removal WG Update (achow101)
<marcofleon>
Erlay WG update is working on fuzzing the txreconciliation class. And I believe we're meeting next week to go over the current open PR #30116. I think it still needs a bit of rework before being fully ready for review
<jonatack>
why not have that discussion at the irc meeting?
<ajonas>
jonatack: I will write that up
<stickies-v>
re wg logistics: wg leaders could also write up their summary just before meeting start (e.g. in a shared doc, or whatever), and then instead of waiting for the summary and for responses, we just have to wait for responses?
ne0h_ has quit [Quit: Leaving]
<sr_gi[m]>
Sorry, looks like my messages didn't go through for the Erlay update
<sipa>
sr_gi[m]: we read you loud and clear now
<jonatack>
(my humble thought is that these open, public IRC meetings are the ideal forum for discussions like that, rather than at a private meeting)
<sr_gi[m]>
We have created a Signal group for those who want to start working/reviewing changes on Erlay, feel free to ping me or Gleb, or I can just paste the group link here, whatever is more convenient
<ajonas>
I led the discussion on the changes to priority projects. I’m happy to speak further with whoever is interested.
<cfields>
hi
<sr_gi[m]>
We are currently working on sims for Erlay plus getting up to speed with the people who have shown interest. Gleb offered some of his time to explain (either in one-on-ones or as a group) what the current merged code looks like, what the dessign is, and what the ongoing PR is about. I'm also happy to do so
<jonatack>
(same for review discussions like that one...why take them to private forums?)
kevkevin has joined #bitcoin-core-dev
<ajonas>
It’s clear they were ineffective and so we discussed changes. There is nothing precluding you from participating in them.
<jonatack>
ajonas: the interesting question is *why* they were ineffective
<jonatack>
imho
<laanwj>
no one knew why, it was just noticed
<ajonas>
Yes. I will write that up as promised above.
<sipa>
We observed that they worked initially, and didn't really work later. Various theories were suggested for why that may be the case, which I'm sure will end up in the meeting notes.
<achow101>
I don't think the why is all that interesting. rather what worked the first time (since everyone seemed to agree the first time was successful) and what can we take from that to do something different
<achow101>
so we have something that works all the time
<bitcoin-git>
[bitcoin] stickies-v opened pull request #31149: tinyformat: enforce compile-time checks for format string literals (master...2024-10/remove-string-format-overload) https://github.com/bitcoin/bitcoin/pull/31149
<jonatack>
one starting point might be to ask, what process initiatives *have* worked long-term
<jonatack>
sustainably
<achow101>
that is what was discussed
Emc99 has joined #bitcoin-core-dev
<achow101>
the reason for discussing at coredev is because it's way easier to do that, and also easier to monopolize 2-3 hrs of people's time to do so
<achow101>
as opposed to irc where people frequently talk simultaneously the typing delay results in some confusing and information loss
<jonatack>
achow101: my opinion is that opening a topic at the open, public weekly meeting is ideal for discussions like that
<achow101>
noted
<laanwj>
also, people tend to attend coredevs in larger numbers than weekly IRC meetings
<achow101>
jonatack: that is why we're talking about it now
<achow101>
but having an initial discussion outside of this meeting makes it easier for most people to be on the same page
<laanwj>
Matt really wants to discuss the mining topic lets go :)
<jonatack>
i wonder if any of the process initiatives have stuck long-term
<achow101>
#topic mining (BlueMatt)
<BlueMatt[m]>
Apologies in advance for the wall of text, I'd tried to have this conversation in a higher-bandwidth way but was rebuffed, so instead we get garbage 🤷♂️
<jonatack>
and if not, consider why that is that case
<BlueMatt[m]>
So I’m really quite confused here, somehow we’ve ended up with Bitcoin Core NIHing a new work providing protocol, except without any of the features of the Sv2-defined one, which is borderline worse than getblocktemplate and where no one who works on mining stuff was consulted on the design of the new protocol.
<BlueMatt[m]>
my understanding of what happened is basically that Sjors was getting zero review and making zero progress on getting the Sv2-defined protocol in core, but there was some feedback that the Sv2 server thing should be in a separate process as a part of the multiprocess transition. This was originally an internal interface question so I pushed back a bit on timeline and practicality but as not a major contributor I don’t really get to
<BlueMatt[m]>
have a role that so that’s the way it went.
<BlueMatt[m]>
After doing initial work for that, Sjors then took stock and realized he's still getting zero review and making zero progress, so instead he/others(?) decided to just make the above interface a public thing, committing to it as a formal public interface with intent for the Sv2 protocol translation thing to be maintained separately.
<BlueMatt[m]>
This is fine, except that, of course, there's absolutely no reason to have an Sv2 protocol translation proxy at all cause now we'd have two protocols and an extra daemon to maintain when we can just skip that, making this new protocol the getblocktemplate de-facto replacement. I don’t have any particular feelings towards the Sv2 specific protocol here, something else which is equivalent would be totally fine, but there are various
<BlueMatt[m]>
goals that went into it that seem to have been totally ignored here.
<BlueMatt[m]>
The whole point of the Sv2 JD server stuff was (a) to be remote-able (TCP + cryptographically authenticated) and (b) to be "push-based" ie let Bitcoin Core decide when to create new block templates and let it immediately push templates when it wants to do so (possibly even before updating the mempool by predicting the next block's contents in advance or in a parallel thread that can run without cs_main). This new protocol is neither,
<BlueMatt[m]>
it is both local-only (and explicitly designed in such a way that you can trivially DoS a node with it) and not push-based.
<achow101>
Sjors does not appear to be here so idk how much discussion is going to happen
<BlueMatt[m]>
The first goal, of course, you can build with a proxy, though to do so now you're back to having to have two daemons to maintain and two protocols which is gonna limit adoption substantially. Ideally the protocol wouldn't be so trivially DoSable so that we can make it public by just using netcat or something which trivially wraps a local connection and provides cryptographic authentication (assuming Bitcoin Core doesn't provide that in
<BlueMatt[m]>
the future which ideally it would even if just not in the first release).
<BlueMatt[m]>
The second goal you can not - if it’s less efficient coming out of Bitcoin Core, no amount of proxying is going to fix that.
<BlueMatt[m]>
It seems that Bitcoin Core took a principled position that the remote-ability of this protocol doesn’t matter (which is a reasonable decision, the remote-ability of work-providing is a somewhat speculative feature, but one I think is worth having a decade down the line), but I’m not really sure why no one who’s spent time on mining work protocols was even consulted on such a major decision. Further, the fact that the one major
<BlueMatt[m]>
efficiency gain that Sv2 was proposing here was thrown out kinda boggles my mind.
<BlueMatt[m]>
Finally, it’s worth noting that getblocktemplate actually has a BIP, but for some reason this new work isn’t even standardized anywhere, which strikes me as strange given Bitcoin core will definitely not be the not server for this so we really can’t just say “the interface is what Bitcoin Core provides”.
<BlueMatt[m]>
the decisions seem to have been much broader than sjors
<BlueMatt[m]>
though I intend to speak to him in atlanta today/tomorrow
<BlueMatt[m]>
apparently he's there
<sipa>
BlueMatt[m]: we did have a long discussion about this topic at coredev
<laanwj>
would have been better to have Sjors here with this topic though
<BlueMatt[m]>
but the situation is quite FUBAR, so its worth highlighting here...
<BlueMatt[m]>
sipa: yes, and I attempted to provide input because it seems weird that these decisions were made to design a whole new protocol without discussing it with anyone who has worked on mining protocol, but was told to eat shit 🤷♂️
<sipa>
BlueMatt[m]: you're not being constructive here
<_aj_>
is there some PR you're referring to?
<BlueMatt[m]>
I'm not sure which PR it was, the protocol is already merged, however
<TheCharlatan>
I don't think this is beyond salvage. All that has happened so far is that a interface that would have been used internally in any case is now exposed. It was not intended to be a protocol, just an interface.
<BlueMatt[m]>
I did open an issue to highlight at least the push-based transition and sjors indicated that that could maybe still be done
<sipa>
i think you're jumping to conclusions that the IPC protocol is (a) immutable (b) intended to be an external protocol
<_aj_>
or #30440 or #30955 maybe?
<laanwj>
the main train of thought is that it was not realistic to have full Sv2 support merged into core, so the second best would be to have a high-performance interface and an external implementation
<BlueMatt[m]>
TheCharlatan: two notes: (a) the interface was not sufficient for use internally, but it sounds like that might change
<BlueMatt[m]>
sipa: no, i was told explicitly it intends to be public now
<laanwj>
if the interface is not sufficient, then that should be improved of course
<BlueMatt[m]>
laanwj: yea, again I don't have any particular desire to see Sv2 merged, but rather the interface needs to be carefully considered
<BlueMatt[m]>
because if its "the public interface" then it *is* a replacement
<achow101>
my recollection is that the sv2 stuff would still live under "Bitcoin Core" but not in the main repo
<sipa>
BlueMatt[m]: some people have opinions, not everyone agrees on all the details, and nothing is set in stone
<laanwj>
do weigh in on interface details then
<achow101>
so having the interface facilitates that
<BlueMatt[m]>
achow101: that's somewhat nonsensical - if bitcoin core is defining a new protocol for work provisioning, there's no reason to have a different one
<BlueMatt[m]>
the immediate short-term issue is #31109
<BlueMatt[m]>
which generally discusses the need to rewrite the protocol
<sipa>
BlueMatt[m]: my personal view is that i'd like a maintained sv2 implementation as part of the bitcoin core organization, and part of its releases, tested against bitcoin... whether or not that's built from the same codebase, whether or not that's written in the same language
<BlueMatt[m]>
but it also almost certainly needs a bip, or at least a specification document in core
<laanwj>
the miners would still want to use Sv2 though? IPC is not something you'd want to offer to the internet, way to large attacks urface
<sipa>
and that the IPC interface evolves from an internally defined to something stable, depending on whether there is interest in external implementations against it
<BlueMatt[m]>
sipa: I don't see why we should have two protocols that do the same thing and (hopefully end up) roughly equivalent?
<BlueMatt[m]>
that seems like a lot of indirection and extra crap everyone needs to implement.
<TheCharlatan>
BlueMatt[m] sure, it will be improved. This was an attempt to get something out something out. From what I gather sjors' current implemetnation would have had the same issue you describe too.
<achow101>
BlueMatt[m]: I don't think that's nonsensical. it's generally the direction we'd like to move towards - interfaces that are used by other parts of Bitcon Core which live in separate repos but under the same org. the interfaces are still largely an internal thing
<sipa>
BlueMatt[m]: please, be constructive
<laanwj>
^^ that
<BlueMatt[m]>
laanwj: ideally the protocol would be restructured to not be so DoS-able :)
<BlueMatt[m]>
which it sounds like might happen on #31109
<achow101>
BlueMatt[m]: because the point is not to have it be publicly exposed that anyone can connect to
<laanwj>
Matt Corallo: sure! it's just that IPC is an interprocess interface, not a remote interface, and will never be
<achow101>
and at least the pr that introduced it is just a refactoring of current stuff to get the ball rolling
<BlueMatt[m]>
achow101: right, like i mentioned in the original wall of text, I didn't weigh in particularly on that decision because that's a bitcoin core decision, not mine. however, it sounds like now the interface intends to be something that is stable/maintained.
<BlueMatt[m]>
at least that's what ive now repeatedly been told.
<sipa>
BlueMatt[m]: again, you are jumping to conclusions
<BlueMatt[m]>
no, I'm repeating what I've been told :)
<sipa>
i'm sure some people hold that view
<achow101>
BlueMatt[m]: stable and maintained != exposed to the internet and therefore needs all the stuff to deal with that
<laanwj>
right
<BlueMatt[m]>
and, more generally, if the IPC protocol *is* reasonably well-maintained such that its compatible-ish across versions then I do think we should kill off the sv2 work protocol thing cause like, why have one?
<achow101>
BlueMatt[m]: ... because it isn't designed as a rpc protocol to be exposed to the internet
<achow101>
it's an ipc protocol
<TheCharlatan>
yes
<BlueMatt[m]>
it doesnt have to be exposed to the internet in v1, of course, but ideally the protocol is structured to support that eventually, cause that's really where it should end up, assuming its set up sensibly
<sipa>
BlueMatt[m]: i disagree
<achow101>
maybe in like 30 years
<laanwj>
that's where Sv2 comes in, as a standard that goes beyond bitcoin core
<BlueMatt[m]>
achow101: i mean if it ends up being a push-based protoocl (post-31109) then it basically is something that could be exposed to the internet trivially
<BlueMatt[m]>
could be done with netcat just copying all the messages from core, even, totally write-only
<laanwj>
our interface is our interface, it just needs to be able to provide what Sv2 needs to use, it isn't supposed to be an actual protocol for outside use!
<tdb3>
hi
<BlueMatt[m]>
my point is that once it supports sv2, it will be basically write-only
<BlueMatt[m]>
so why not just let it be ~write-only (with one message to get the coinbase tx size)
<laanwj>
IPC will also provide the interface for wallet, GUI,and so on, all semi-internal
<BlueMatt[m]>
because once we have that exposing the same protocol with netcat is trivial :)
<sipa>
i would have preferred sv2 directly in bitcoin-core-the-bitcoind-binary, but i understand the objections many contributors have raised; i think for supporting the mining ecosystem, having an external binary/process that implements sv2 that speaks this IPC, and talks the well-defined Sv2 protocol, and is released-with, and tested-against every version for bitcoind
<BlueMatt[m]>
I also generally think miners will refuse to use that - lower latency is lower latency, to them
<laanwj>
there's just such a severe lack of reviewers interested in any mining stuff, if there were more, then maybe it would have been realistic to have Sv2 directly in core
<BlueMatt[m]>
even though I think they're wrong and simplistic...
<BlueMatt[m]>
and, again, I don't feel strongly about sv2 actually happening here, this protocol in sv2 is *trivial* and doing anything equivalent to it is totally fine by me
<sipa>
is effectively the same thing, and offers some advantages like process separatations (bitcoind code reviewers don't need to review the sv2 protocol implementation for "will this not crash bitcoind"
<laanwj>
Sjors has been mostly working on his own on this, if not for him, then there would be no progress on it at all likely
<sipa>
BlueMatt[m]: it'd be helpful if you'd review the code then
<BlueMatt[m]>
I don't believe I'm qualified for this anymore, at least not C++, but I did go and review the mining protocol to start a conversation about that, yes.
<BlueMatt[m]>
at least in structure
<BlueMatt[m]>
I also can't say I'm super jazzed about the entire bitcoin ecosystem having to support capnproto, but hey whatever, not really a huge deal (is it trivial to write a parser for it?)
<laanwj>
that's one reason why he'd like the Sv2 server to be in rust
<sipa>
BlueMatt[m]: ffs, it does not
<laanwj>
so all the people who want to avoid c++ can finally look at something :)
<BlueMatt[m]>
laanwj: ha
<sipa>
BlueMatt[m]: capnproto and IPC is how bitcoind and the sv2 implementation can talk
<achow101>
BlueMatt[m]: No one except you wants this interface to be publicly exposed
<achow101>
and no one is intending on doing that
<sipa>
it could be implemented in rust even (which i've heard some ideas about)
<BlueMatt[m]>
achow101: that's not what I was told? sjors told me explicitly it intends to be documented/public
<laanwj>
documented, yes
<achow101>
BlueMatt[m]: *publicly exposed to the internet
<laanwj>
exposed, no
<BlueMatt[m]>
but, again, if the interface exists and is semi-stable, people *will* use that
<BlueMatt[m]>
right, so exposed is a separate question
<sipa>
BlueMatt[m]: well Sjors[m] isn't here
<BlueMatt[m]>
once this becomes the standard mining protocol we'll presumably drop the sv2 protocol
<laanwj>
we'd like to document it for people that want to use it in an inter-process way on the same machine but not use the bitcoin core tool, for some reason
<sipa>
i really don't see why
<cfields>
how would it even be exposed?
<TheCharlatan>
exposed meaning there is a unix socket you can connect to on the same system. Not a public port exposed to the internet.
<BlueMatt[m]>
and then suggest people expose this by netcat, at least once its write-only and pretty safe to do :)
<sipa>
i think sjors may have had the impression that stable-ipc-talking-to-sv2-binary was his only way forward; i hope he's wrong
<achow101>
so we should make it not write-only to prevent that :)
<BlueMatt[m]>
sipa: I do believe that was his impression, i have no idea if he's right, but I'm not sure it matters?
<BlueMatt[m]>
achow101: I don't think any sensible work-providing protocol would be anything but :p
<laanwj>
it's not write-only, capnproto IPC just isn't suitable for that kind of thing, it has a large meta-attack surface wrt handles and such, which simplifies having high performance local interfaces but isn't suitable for internet usage
<BlueMatt[m]>
one message to get the coinbase tx length and then write-only
<sipa>
BlueMatt[m]: i don't think it matters, because i feel that the direction that things have been going right now is helpful for multiple possible outcomes, and furthermore, it is making progress
<BlueMatt[m]>
laanwj: the current one isn't, but it needs to be rewritten for efficiency anyway.
<achow101>
also, this interfaces thing likely would have happened anyways with the work on multiprocess
<BlueMatt[m]>
sipa: right, that's basically my impression.
<achow101>
since that is the direction the project is going in
<laanwj>
wait, now you want to rewrite capnproto? i...
<BlueMatt[m]>
I'm not complaining about the fact that we have a different mining protocol, I don't care, I'm highlighting that we need to treat it as a Mining Protocol
<BlueMatt[m]>
because that's what it is
<achow101>
perhaps we should revisit this discussion when Sjors is here and the meeting notes are published
<BlueMatt[m]>
irrespective of whether we want it to be or not
<laanwj>
it's an internal interface to facilitate mining protocol implementations
<laanwj>
not a Mining Protocol
<BlueMatt[m]>
why would a pool not connect directly to it?
<sipa>
BlueMatt[m]: i really don't think it's helpful that you keep asserting that this is a mining protocol, despite many people here telling you that their view is that it won't be?
<BlueMatt[m]>
I don't see why anyone would use a proxy to translate one protocol into an equivalent one?
<laanwj>
i don't think it's useful to discuss this further without Sjors here
<BlueMatt[m]>
we can discuss it again next week when sjors is back
<laanwj>
yea
<achow101>
Any other topics to discuss?
<jonatack>
Sjors[m]: you can add me to your working group
marcofleon has quit [Quit: Client closed]
<jonatack>
Helpful discussion (for my understanding at least), thank you BlueMatt[m] and everyone for doing it here
marcofleon has joined #bitcoin-core-dev
<achow101>
#endmeeting
<cfields>
It's not even just a mining interface. the ipc work is attempting to make it easier to construct 3rd-party-ish downstreams of bitcoind. Mining just happens to be a consumer that makes sense for it.
<cfields>
gui/wallet as other obvious examples.
Emc99 has quit [Quit: Client closed]
<bitcoin-git>
[bitcoin] maflcko opened pull request #31150: util: Treat Assume as Assert when evaluating at compile-time (master...2410-assume-stronger) https://github.com/bitcoin/bitcoin/pull/31150
<BlueMatt[m]>
<cfields> "It's not even just a mining..." <- Right, but easy enough to put mining on a separate socket, presumably. I’m noting that this is how people are going to use it, cause mining pools panic about latency and an extra proxy is obviously latency that they don’t want, so they’ll connect directly…we don’t get to control what they do, sadly.
<BlueMatt[m]>
Ah I think some of the confusion was that sv2 is all about being exposed to the internet. It’s not, and that’s a very speculative feature that people certainly aren’t goona use any time soon. I think it’s important, but very forward-looking. Thus a socket interface is what people want, at least for the next few years.
Guest1 has quit [Ping timeout: 256 seconds]
<gmaxwell>
Any kind of external protocol translator will just be a failure point. like the situation right now where mining devices don't speak GBT, so to solo mine you need a proxy and the software documented for that is luke's BFG miner which (last I checked) doesn't compile without considerable work (including making patches and hunting down archived copies of stuff), won't let you mine to anything
<gmaxwell>
except a p2pkh address, etc. So why would people using one instead of writing their mining infrastrcture against the bitcoin core IPC, skipping a step, source of failures, source of latency?
<gmaxwell>
I had a friend ask me for help setting up solo mining a couple months ago and it basically took me a full day of work to make it happen, it's absurd and really demoralized me about the state of things.
<gmaxwell>
I may have contributed to Matt's rampage above by venting to him about this expirence. (well it started off with a question of 'hey how do I use this sv2 stuff that you've been working on for something like 5 years now'
<bitcoin-git>
bitcoin/master dd92911 merge-script: Merge bitcoin/bitcoin#31148: ci: display logs of failed unit tests automat...
<bitcoin-git>
[bitcoin] fanquake merged pull request #31148: ci: display logs of failed unit tests automatically (master...2024_ci_test_output) https://github.com/bitcoin/bitcoin/pull/31148
preimage has joined #bitcoin-core-dev
brunoerg has quit []
Guest45 has joined #bitcoin-core-dev
zeropoint has joined #bitcoin-core-dev
<laanwj>
if that isn't important, and one could remove all the "internet hardening" features from Sv2 and it'd also be a lot more straightforward to implement in core, i guess
<laanwj>
from what i understood the main worry is having to maintain another P2P-port like interface, with DoS robustness, encryption, etc etc
<laanwj>
so yes if that is a misunderstanding it's one that's already held up a lot of progress
<sipa>
not DoS robustness, but i have heard the two other parts as arguments: the desire to refactor existing P2P code stuff so it could be reused for Sv2 as well was something not everyone wanted, and all the code for implementing sv2's encryption feeling unnecessary
<laanwj>
the goal here is to add a mining interface with a s little impact core as possible (in lines of codes, needed refactors, maintenance), and multiprocess is a way to do this
<sipa>
gmaxwell: FWIW, my view is that whatever we end up with is something that gets tested as part of Bitcoin Core's release process
<sipa>
which i think makes sense if we see this IPC mechanism the way it is intended, as an internal interface between different parts of the same software
pablomartin has quit [Ping timeout: 265 seconds]
<sipa>
but also, this doesn't need to be set in stone, and the fact that it is making progress at all is far better than the opposite; the outcome could be miners/Sv2 stacks implementing this IPC mechanism directly; it could be a Sv2 sidecar binary built from and shipped with bitcoin core; it could even be something like that for some time, but eventually merged back directly
<laanwj>
right just that it's a separate process communicating through ipc doesn't mean it can't be started automatically
cornfeedhobo has joined #bitcoin-core-dev
pablomartin has joined #bitcoin-core-dev
<BlueMatt[m]>
<laanwj> "if that isn't important, and one..." <- I think it’s both: IMO it’s a useful hedge for Bitcoin if the mining protocol people use is trivially moved to running over the wire (with cryptographic authentication), but also that no one is going to use it in the short term- it might suddenly become important overnight at some point but it’s hopefully not soon.
<laanwj>
that's the problem then too... it's hard to warrant implementing something whose expectations are so unclear
<BlueMatt[m]>
Can’t say I disagree, hence my stance that not having it initially is fine but ideally the protocol would trivially support it. Luckily that goal aligns with other important goals around performance
<laanwj>
in any case, i hope Sjors doesn't get demotivated
<gmaxwell>
I don't think it matters to me if there is some process seperation thing going on (though the latency of the extra serialization round trip might be unfortunate, but thats a benchmarking question). Just that you know it ought to be pratical to mine bitcoin in 'lottery mode' without depending on a trusted third party... and by pratical I mean without the side quest through broken mazes of
<gmaxwell>
missing gitlab repositories and code that doesn't compile. The obvious thing to do would be to implement stratum v1 mining support, but my understanding is that this was unattractive for varrious reasons and sv2 was intended to be designed with input that made it both meet mining needs *and* be supportable by node software... but the latter part may have been a failure?
sdaftuar1 has joined #bitcoin-core-dev
<sdaftuar1>
exit
sdaftuar1 has quit [Client Quit]
<sipa>
gmaxwell: i agree with that
sdaftuar has quit [Quit: WeeChat 3.0]
<achow101>
anyone else finding that running ctest after compiling with clang and Debug mode takes a significantly longer time recently?
<achow101>
seems to be stuck on Test #6: tests for 10 minutes longer than the next longest test
<darosior>
For the assumeutxo background sync, do we disable assumevalid?
<bitcoin-git>
[bitcoin] jonatack closed pull request #31135: rpc, cli: return "verificationprogress" of 1 when up to date (master...2024-10-verification-progress) https://github.com/bitcoin/bitcoin/pull/31135
Talkless has quit [Remote host closed the connection]
<instagibbs>
and sdaftuar I think you're right, the wait for trimming thing isn't a "safety" thing but an ergonomics thing. Funnily enough, if you write the test case so the "final" parent would be evicted, it actually still succeeds since it's a reconsiderable failure
<maflcko>
achow101: Interesting. If it happened recently, can you bisect?
Guest8 has joined #bitcoin-core-dev
Guest8 has quit [Client Quit]
<achow101>
Will try
___nick___ has quit [Ping timeout: 264 seconds]
jespada has quit [Ping timeout: 252 seconds]
<gmaxwell>
darosior: I don't have an answer to your question, but it would be interesting in general for there to be an additional validation level that indicates if signatures have been checked or not, seperately from doing the background sync in two passes being potentially interesting, a validation level would make batch validation more effective by being able to validate several blocks as one batch.
<darosior>
Yes. In fact i think it would be interesting to do check the signatures after IBD, akin to how assumeutxo performs background sync.
<darosior>
(Which is what triggered my question, which i should probably answer by myself)
<sipa>
not needing to ramp up and ramp down the signature validation threads may be a significant speedup
<sipa>
even without batch validation
<gmaxwell>
for long ago blocks like 2012-2015 not having the concurrency slog may help a lot.
<gmaxwell>
darosior: exactly
<darosior>
Because as it is i think that the `-assumevalid` model is less "trustless" than assumeutxo, which eventually verifies everything.
<gmaxwell>
darosior: I vaguely recall sipa and I looking into this before and maybe there wasn't enough space in the stored data to store an extra flag, so it would require an index disk format change, but I could be imagining it.
<sipa>
pretty sure there is plenty of space, but it may require a disk format change anyway for compatibility reasons
<gmaxwell>
darosior: six one way half a dozen another, in the sense that assumevalid has an extremely good review model while assumeutxo doesn't... and also the latter seems extremely likely to evolve into something where no one checks the history (regardless of what we think about that)
<darosior>
So i just checked and since assumevalid is a parameter of the chainstate manager, it appears that it is also enabled for the background sync.
<gmaxwell>
sipa: a format change that just prevents downgrading but is totally inplace is way better than one that requires a rewrite of all the data. :)
<sipa>
gmaxwell: absolutely
<sipa>
well one reason for the background sync to exist is to make sure that the full sync data remains available, which is a concern that doesn't exist with assumevalid
<sipa>
but with the background sync existing anyway, there is no reason why it couldn't also validate signatures i think
<sipa>
(of course, absent a assumeutxo-fetching P2P extension, it's unclear to me how many people will actually use assumeutxo...)
<darosior>
gmaxwell: yes, i was considering this from the angle of "if assumevalid was skipping all checks (including whether inputs exist) instead of just the script checks, would it be a meaningfully different trust model?" and i don't think it would be. But it would also doesn't bring much of a gain for us, since you still need to update the UTxO set
<darosior>
whether or not you check if inputs exist and are yet-unspent. And i agree that the assumeutxo format in itself brings other concerns.
<gmaxwell>
darosior: the thing with assumevalid is that to review its accuracy you just need to say "is this block in my chain or not" and if not sound the alarm, and if the software isn't truthworthy enough for *that* than all bets are off since it's easy to bugdoor to disable validation entirely. ... also any attack requires a significant amount of mining. I agree if it skipped other stuff it wouldn't
<gmaxwell>
be worse, but skipping other stuff would be pointless for it.
<darosior>
Yes.
jirijakes has quit [Ping timeout: 255 seconds]
<sipa>
prior to the assumevalid point, we could even process blocks out of order, by writing a "negative" entry in the UTXO set (which is erased when its creation is encountered)... this is a quite a bit harder if signature checks are needed, because you'd need to put all the relevant information from the spending tx in the UTXO set
<sipa>
reorgs and bip30 shenanigans complicate this further
<gmaxwell>
the other deal with the sig flag is that letting signature validation run async would greatly improve concurrency, like right now during sync you can see slogs of {network, cpu, disk} each operating one at a time and waiting for each other.
<sipa>
indeed
kevkevin has quit [Remote host closed the connection]
kevkevin has joined #bitcoin-core-dev
<gmaxwell>
(also for bystanders, even though modern blocks have lots of transactions making it seem like there isn't too much to be gained by validing multiple at once, I think that's less true once you introduce batch validation... as there are good gains from batching up to over 100 signatures per batch... so like 8000 txins sounds like enough for concurrency even on a 384 thread computer... but against
<gmaxwell>
384 threads that only allows batches of size 20... and running just 20 operations and then synchronizing threads means a lot of parallelism loss.
<gmaxwell>
)
kevkevin has quit [Remote host closed the connection]