<achow101>
Current backports PR is #30959. Since the 2 fixes being backported are trivial, I'm plannin on skipping rc3 and tagging final once that is merged
<fjahr>
Just a quick PSA: I have moved all my ASMap related repos to a dedicated org (https://github.com/asmap) to make collaboration easier. Old links should still work via redirect but if you see an outdated link please update because GH says they don't maintain those redirects forever.
<willcl-ark>
(reading first bit of that): perhaps we should discuss macos notarization? i.e. #15774
<achow101>
fjahr: good to know. IIRC github does keep redirects for a long time, at least I've had a redirect that's been working for 5 years now
<achow101>
#topic MacOS notarization
<achow101>
willcl-ark: I don't think there's really been a plan yet?
<achow101>
but it does seem like we should probably figure it out for the next release
<fjahr>
achow101: yeah, that's just what I read, I guess they want to have the option to remove them at some point
<sipa>
if macos is phoning home regardless of stapling/notarization, there seems to be little downside to doing it
<willcl-ark>
Exactly THe current requirement is that a user must generate a codesigning cert, then codesign, and remove the quarantine bit. We can't really get around it AFAIK
<Chris_Stewart_5>
Seems like a note about this in the release notes would be a good first step? I've run into this multiple times at this point and its a PITA finding the exact commands that need to be run to make bitcoind work on mac
<Chris_Stewart_5>
a quick glance of the 28.0 release notes doesn't contain anything AFAICT. Seems like putting this in the 'How to upgrade' section would make sense? :man_shrugging:
<gmaxwell>
If OSX phones home the software users run it might be worth a persistant note on that too in the docs for running on OSX.
<achow101>
I've moved #15774 to the 29.0 milestone
<vasild>
well, if macos is phoning home and bitcoind does not work on macos, then even better. Why do we need to do anything?
<Chris_Stewart_5>
hebasto: it does, but where are the commands? How are people suppose to find them?
<emzy>
AFAIK it is this: xattr -rd com.apple.quarantine /path/to/applicationname.app
<willcl-ark>
fanquake: no, I'm pretty sure the self signing and removing quarantine bit still works?
<Chris_Stewart_5>
to be clear, this is probably a stop gap, things should probably just work :tm: ? But giving people instructions as a workaround for now seems appropriate?
<sipa>
Chris_Stewart_5: agreed
<achow101>
Chris_Stewart_5: yes
<fanquake>
willcl-ark: not sure, but anyone using bitcoin-qt, which is primarily what we ship on macos, isn't doing any of that
<willcl-ark>
fanquake: agree
<sipa>
vasild: the world would be a lot better place if all humans were reasonable, but if that were the case, we probably wouldn't need bitcoin at all?
<willcl-ark>
(but I have for sure run bitcoind 28rc2 after self-signing)
<vasild>
sipa: yeah, I was half-joking ;-)
<sipa>
to self-sign, you need a self-signing certificate? can you do that without apple's involvement?
<achow101>
Any other topics to discuss?
<sipa>
are there workarounds besides that?
<achow101>
sipa: there's "ad-hoc signing" which doesn't actually require a certificate
<willcl-ark>
sipa: not that I'm aware of. {Perhaps disable the entire SIP mechanism?
<Chris_Stewart_5>
sipa: No need to involve apple, you self sign and the cert gets stored locally iirc
<glozow>
re topics, I was wondering about cluster mempool / TxGraph status? If sipa has an update on that?
<sipa>
glozow: happy to give an update
<achow101>
#topic cluster mempool update
<glozow>
:D
<sipa>
so, we're currently imagining something like 3 layers for the cluster mempool code
<Chris_Stewart_5>
One last comment on the notarization problem on mac, here is a bitcoin stack exchange question detailing the problem and solution: https://bitcoin.stackexchange.com/a/117101
<gmaxwell>
vasild: considering how easy it is to enumerate people running bitcoin from the p2p network the fact that it phones home isn't hard incompatible with bitcoin I think. But perhaps it would be a very unwelcome surprise to people using bitcoin behind tor.
<sipa>
* the bottom layer (depgraph) is pretty much done, apart from 30857 (which i don't plan to change anymore barring significant review comments). It deals with individual transaction clusters in a very abstract way, but contains all the computationally-hard stuff.
<vasild>
gmaxwell: yeah, and it depends on what it phones home - maybe it would automatically backup important files to the cloud (like wallet.dat). A friend of mine complained that some anti virus moved his wallet.dat (!) I wouldn't be surprised if it sent it to the anti virus company for an analysis, if it treated it as a virus.
<sipa>
* the middle layer (txgraph) is sort of a barebones mempool, knowing just about all transaction fees/sizes and dependencies between them (but has no concept of txids, outputs, inputs, prioritization, ...), clusters of them, and linearizations for them...; i have been working on fleshing the design for this out, but i don't have anything to show yet, sorry
<sipa>
txgraph would also have a notion of "changesets", proposed sets of transaction additions/removals/dependencies to add to the mempool, so a rbfs can be staged (create a changeset for a proposed RBF, inspect the feerate diagram changes it would make, and then decide to throw it away or commit it)
<sipa>
* the top layer is the mempool/validation code, which would use txgraph but add actual transactions, policy rules, validation
<sipa>
the point of txgraph is that being more abstract it can be tested in isolation much more easily
<sipa>
suhas' current PR (28676) contains a prototype for txgraph, but i'm working on rewriting it as a separate PR, after which 28676 would be rebased on that
<glozow>
This sounds really cool!
<glozow>
Thanks for the update sipa
<tdb3>
thanks sipa!
<achow101>
anything else to discuss?
<Chris_Stewart_5>
sipa: Perhaps cart before the horse, but what do we think the deployment story looks like?
<Chris_Stewart_5>
last I heard, the idea would be the totally replace existing mempool code with the new cluster mempool code, is that still the plan (presumably in 29.0)?
<sipa>
Chris_Stewart_5: i think everything up to txgraph (which may well end up becoming multiple PRs) can be staged in, as in merged without actually changing any observable behavior, as it's just new data structures and algorithms that can be tested in isolation
<sipa>
but the introduction of cluster-based mempool will be a hard switch; i don't think we realistically want both the old and new code ever simultaneously in production (it'd involve the worst of both worlds: the memory and CPU costs related to having both, plus the policy rules that are necessitated by both).
<sipa>
i could imagine the introduction of cluster mempool, and the removal of the old mempool data structures to be in separate PRs only if the intent is for both to go in in the same release.
<Chris_Stewart_5>
I guess I was thinking more along the lines of a -clustermempool flag in 29.0 and then assuming everything goes good, default to -clustermempool in 30.0 and remove old code in 31.0? Perhaps this a bit too conservative though?
<Chris_Stewart_5>
Is this written about anywhere why its not possible? Or just have to read through code?
<sipa>
i just told you :)
<sipa>
it would involve the worst of both worlds
<Chris_Stewart_5>
Yeah, and I guess I assumed only 1 of the mempool implementations would be running (-clustermempool or legacy mempool). Not both.
<sipa>
they're just data structures that need to be maintained - if they're compiled in, they will be used
Emc99 has quit [Ping timeout: 256 seconds]
<sipa>
at the very least you'd get the memory usage from both
<sipa>
it's not like we're creating a completely new mempool and can decide at runtime which of the two you're using
<_aj_>
if you're nervous about the new mempool code in 29.x you'd just keep running 28.x anyway?
<Chris_Stewart_5>
Runtime performance and code maintenance are different topics. I agree there will be maintenance burden for awhile. Idk if i'm so convinced on runtime performance, but you have more experience than i do on the topic so i'll defer to you
<sipa>
we do have simulations based on replays of all mempool data to get an idea of what would be impacted by the policy changes that cluster mempool entails
<Chris_Stewart_5>
_aj_: Yeah, and then we give people another reason not to upgrade with the latest and greatest stuff and get them stuck on 28.x.
Wronsk has joined #bitcoin-core-dev
<_aj_>
Chris_Stewart_5: if they don't upgrade from 28.x until 30.x is out, that's already fine and supported
<Chris_Stewart_5>
Any way, perhaps i'm concern trolling at this point so i'll keep quiet. I am excited about clustermempool and think its great!
<sipa>
Chris_Stewart_5: i think the biggest issue is the policy rules... the current mempool has certain policy rules that are necessitated by its implementation (primarily the ancestor and descendant set limits); cluster mempool has different policy rules (cluster size limit, which isn't exactly the same as ancestor or descendant limits, but supports higher numbers)
<sipa>
if we keep the old mempool data structures, we need to keep the old policy rules
<sipa>
if we introduce the new mempool data structures, we need to introduce the new policy rules
<sipa>
so having a release that supports both would effectively entail having the combination of both policy rules (neither ancestor/descendant limits can be violated, nor cluster size limits).
<glozow>
I imagine it would be even more invasive to add code everywhere (validation, policy, mining, etc) to have 2 mempools. And they'd have different transactions in them... sounds like a nightmare
<glozow>
(And I can't think of another way to support both)
<Chris_Stewart_5>
I'll think about this more. We can move on
<sipa>
Chris_Stewart_5: yeah i understand the concern; it will be a nontrivial and abrubt change - but i don't think the alternative is very realistic
<vasild>
"I guess I assumed only 1 of the mempool implementations would be running"
<sipa>
a compile-time switch between them may be vaguely possible
<sipa>
but even that is pretty hard
<achow101>
anything else to discuss?
<tdb3>
Sounds like the probability of bugs would be higher in a conjoined/transition implementation than with a clean update. And an end goal is minimizing bugs
<sipa>
Chris_Stewart_5: re apple... if you self-sign, do you still have the phoning home?
<b10c>
would love to have a few more eyes on #25832. I've been using the connection tracepoints for >2 years now. They've helped learning out about LinkingLion and us dropping connections due to high CPU usage elsewhere
<b10c>
I'm frequently rebasing these commits on master and e.g. RCs that I monitor - hope to run plain master at some point
twistedline has quit [Ping timeout: 276 seconds]
<Chris_Stewart_5>
sipa: i'm not sure what you mean by phoning home, and I'll need to investigate regardless. Do you mean contacting apple ?
<sipa>
Chris_Stewart_5: yes
<Chris_Stewart_5>
i don't believe so, but i'm going to go investigate now. Be back in awhile with a PR
<fanquake>
by default any codesigning may involve timestamping, which i'd assume apple would use it's own servers for
<sipa>
my understanding is that we've previously held off on doing these things because apple would phone home to verify the certificate in some way or so
<achow101>
ad-hoc signing doesn't actually sign though
<achow101>
it actually tells gatekeeper that the binary is safe to not verify
<sipa>
but given recent reports that this contacting of apple servers happens regardless, it may be time to bite the bullet
<fanquake>
has anyone checked if the codesigning tool actually ships on a vanilla, non-dev mac
<fanquake>
or will we also need to tell people to download and install the command line tools (from apple)
<fanquake>
to actually do any signing in the first place
<achow101>
fanquake: i believe it does as it's part of the security utils. haven't checked on a mac though
<sipa>
Chris_Stewart_5: i ask, because if somehow the phoning home can be prevented by self-signing, then perhaps self-signing should remain the primary recommendation, even if we do the stapling
twistedline has joined #bitcoin-core-dev
<fanquake>
Can someone share whatever command they are using for adhoc codesigning? I just downloaded bins from bitcoincore.org, ran codesign --force --options runtime -s - bitcoind, which shows, Signature=adhoc, but the bins still don't run
twistedline has quit [Ping timeout: 260 seconds]
twistedline has joined #bitcoin-core-dev
Wronsk has quit [Remote host closed the connection]
twistedline has quit [Ping timeout: 260 seconds]
twistedline has joined #bitcoin-core-dev
Emc99 has joined #bitcoin-core-dev
Emc99 has quit [Client Quit]
<bitcoin-git>
[bitcoin] Christewart opened pull request #30982: docs: Add instructions on how to self-sign bitcoin-core binaries for macOS (master...2024-09-26-selfsign-mac-instructions) https://github.com/bitcoin/bitcoin/pull/30982
twistedline has quit [Ping timeout: 244 seconds]
<Chris_Stewart_5>
fanquake: ^ is what works for me. Give it a try. It appears that the xattr -d com.apple.quarantine is only required if you download via a web browser (rather than command line) AFAICT
twistedline has joined #bitcoin-core-dev
<Chris_Stewart_5>
and i don't believe it requires phoning home. I disconnected from my network connection and was still able to sign and run commands like ./bitcoin-cli --help
<Chris_Stewart_5>
fanquake: Also according to chatGPT 'codesign' ships by default on macOS ;).
<fanquake>
Chris_Stewart_5: cool. Note that we'll want to inline the instructions into the readme, rather than linking to one thing, that then links to something else
<fanquake>
as far as I'm aware, it's also not required to create a cert, as claimed in the second linked post
<Chris_Stewart_5>
I"ll just remove the link as I don't think it provides much value
szkl has quit [Quit: Connection closed for inactivity]