< wumpus>
yeah I think promising people a bounty for implementing something is okay
< wumpus>
given the amount of work involved $300 is not that much, but it does show people want this particular feature badly, and actually offering money is much better than the usual entitled "when????" github post
< jonasschnelli>
indeed
< jonasschnelli>
I vaguely remember getting a BTC bounty when I implemented the initial HD wallets patch
< jonasschnelli>
Can't remember the website though
< bitcoin-git>
[bitcoin] stackman27 opened pull request #20874: test: Run mempool_limit.py even with wallet disabled (master...diswallet_mempoollimit) https://github.com/bitcoin/bitcoin/pull/20874
< jonatack>
I received a (non-trivial) bounty recently for some larger PR reviewing done in October just before the feature freeze.
< jonatack>
jonasschnelli: maybe from bitcoin acks
< jonasschnelli>
jonatack: nice. No mine was before bitcoinacks existed
< wumpus>
jonatack: something that surprised me (but no big deal) is that "-netinfo help" only works when there is a remote server, even though the help message is entirely formatted locally
< jonatack>
wumpus: ah! i didn't think of trying that. you are referring to "error: Could not connect to the server 127.0.0.1:8332 Make sure the bitcoind server is running and that you are connecting to the correct RPC port."?
< jonatack>
i'll see if can move the help check earlier
< wumpus>
jonatack: yes, that, while testing the help message I did somehow not expect it to need the RPC server running
< wumpus>
jonatack: I also noticed that "netinfo 5" behaves like "netinfo 0", instead of "netinfo 4"
< jonatack>
wumpus: good point, maybe invalid integers should raise too
< jonatack>
yes, just as the summary help doesn't need the server, e.g. ./src/bitcoin-cli -h | grep -A4 netinfo
< wumpus>
jonatack: the behavior I'd personally expect is either a) higher, not (yet?) defined levels act as the highest defined level b) it raises an error
< wumpus>
yes maybe an explicit error is best
< wumpus>
I think something we need to expicitly warn against is using this output in scripts in favor of parsing JSON, it's not a stable interface, maybe we do this already
< wumpus>
(the same we did for readelf, human readable formats are for humans not for machine parsing)
< jonatack>
i thought it was understood that software clients shouldn't depend on client-side cli commands, e.g. instead of -netinfo they should consume getpeerinfo and getnetworkinfo that are subject to API stability constraints
< jonatack>
hm, git grepping for "unstable\| stable" doesn't turn up much explanation on this. Maybe it's not obvious. I'll look at clarifying.
< wumpus>
I think it's generally understoof, but it doesn't hurt to explicitly mention it, a new user might not be aware what the division between client/server side is exactly
< bitcoin-git>
[bitcoin] MarcoFalke merged pull request #20789: fuzz: Rework strong and weak net enum fuzzing (master...2012-fuzzRefactor) https://github.com/bitcoin/bitcoin/pull/20789
< bitcoin-git>
[bitcoin] jonatack opened pull request #20877: netinfo: user help and argument parsing improvements (master...netinfo-help-follow-ups) https://github.com/bitcoin/bitcoin/pull/20877
< wumpus>
hebasto: #18710 is great, it needs very careful review due to script validation being consensus critical, but good to get rid of another boost::thread_group use (on of the last I think?)
< wumpus>
let's at least wait a week or two before tagging final, I'm sure if people start testing issues will come up
< MarcoFalke>
sipa: Or none of them
< jnewbery>
wumpus: can we move #20557 to high priority bug fixes? PR #16702 broke some aspects of addrman deserialization and it'd be nice to fix them
< gribble>
https://github.com/bitcoin/bitcoin/issues/16702 | p2p: supplying and using asmap to improve IP bucketing in addrman by naumenkogs · Pull Request #16702 · bitcoin/bitcoin · GitHub
< sipa>
not being able to disconnect misbehaving tor nodes seems like something worth addressing
< wumpus>
yes, rc4 and rc5 were doc only
< wumpus>
apart from the torv2->torv3 signet seed change
< wumpus>
MarcoFalke: ofc, assuming nothing comes up with rc5
< wumpus>
so we could make it depend on that, if a rc6 is needed for another reason, include, say, 20852
< wumpus>
ok, this starts to be a monologue, any other topics?
< MarcoFalke>
sipa: *outbound tor nodes
< MarcoFalke>
sipa: Delaying the release will also delay already included bugfixes that exist in previous releases
< MarcoFalke>
So I think it is a trade-off
< sipa>
?
< wumpus>
I don't think in itself it warrants delaying the release
< sipa>
ok
< wumpus>
of course we could merge it and do rc6 right now but that'd not be very mindful of people that just started testing rc5, better to stick with a rc per week at most
< MarcoFalke>
rc5 pretty much only needs testing on macOS
< MarcoFalke>
apart from the signature, nothing changed code-wise
< luke-jr>
could always just go 0.21.1 sooner too
< wumpus>
if there's no other topics, that's a short first meeting of the year
< wumpus>
MarcoFalke: luke-jr: agree
< MarcoFalke>
luke-jr: 0.21.0.1 ;)
< jonasschnelli>
<MarcoFalke> rc5 pretty much only needs testing on macOS
< jonasschnelli>
^ why?
< MarcoFalke>
jonasschnelli: Only documentation fixups were merged
< luke-jr>
MarcoFalke: no reason for that mess :P
< MarcoFalke>
(compared to rc3)
< jonasschnelli>
It was testable on macOS before rc5
< luke-jr>
oh, I have a possible topic
< luke-jr>
Debian wants to include bitcoin core again
< sipa>
MarcoFalke: i don't think that's a meaningful difference
< sipa>
it's not like you're going to review all the things that get changed when updating
< sipa>
also, auto-update can be enabled
< luke-jr>
If we are auto-updating via official Snaps, I don't think we can make that objection
< wumpus>
right, the point is that bitcoin needs *special* review compared to upgrading other software
< MarcoFalke>
I do review them (at least of the important packages)
< luke-jr>
MarcoFalke: can you confirm the Snap situation?
< MarcoFalke>
luke-jr: Yes, there is an auto-update footgun
< MarcoFalke>
I think it is optional
< MarcoFalke>
Also, there are tracks for previous releaes
< sipa>
regardless, i don't think we're ready for 3-year maintenance
< MarcoFalke>
So even with auto-update you can say "only auto update on the 0.19 branch"
< jonasschnelli>
yes
< MarcoFalke>
(no one uses the tracks, though)
< wumpus>
I definitely have no interest in maintaining a 3-year old release
< jonasschnelli>
plus core is very easy to "install" on most systems thanks to the static binaries
< wumpus>
maintaining he releases we do is already overwhelming enough sometimes
< luke-jr>
anyway, I guess it's moot unless someone wants to maintain for 3 years..
< MarcoFalke>
sipa: Creating a docker image should be trivial. Not sure if it is deterministic, though.
< luke-jr>
I've done it before, but having no funding, I'm not too inclinded to repeat it
< jonasschnelli>
no fun(ding)
< wumpus>
it's definitely a job that would require funding, no one is going to do it for fun :)
< wumpus>
if there are really companies that want to run a three-year old stable, make them fund it
< sipa>
MarcoFalke: i read that docker images can be identified by their hash, so if creating one is deterministic, that may be something people are interested in having
< sipa>
(i've never used docker myself, though)
< wumpus>
okok docker
< wumpus>
#topic Docker image generation in gitian
< core-meetingbot>
topic: Docker image generation in gitian
< MarcoFalke>
sipa: me neither
< achow101>
in theory it is deterministic because you can specify parent image hashes
< jonasschnelli>
sipa: why inside gitian? What are the benefits?
< achow101>
in practice, I've had no such luck
< MarcoFalke>
jonasschnelli: For release binaries
< sipa>
jonasschnelli: otherwise we're relying on external people to do it?
< MarcoFalke>
We could also just include a docker script
< jonasschnelli>
Wait? For uploading the gitian built deterministic release binary?
< sipa>
jonasschnelli: the idea is that we could publish a docker image hash, together with the release binaries & their hashes
< MarcoFalke>
no, in parallel to the snap package, also offer a docker package
< sipa>
jonasschnelli: and if that hash if verified through gitian, it means it has the same standard of review and auditability as the rest of the release process
< jonasschnelli>
okay.. I see.
< sipa>
i'm just mentioning this because i saw https://github.com/bitcoin-core/packaging/issues/55, and it seems doing it inside gitian is an almost trivial change for us, apart from some initial work to set up the scripts
< sipa>
jamesob seems to maintain a script for building docker images?
< MarcoFalke>
I just feel that we need a docker expert to pull this off. Also, it would be sad if that blocked progress on guix
< luke-jr>
hmm, true
< wumpus>
so the question then would be: for what platforms? we can't quite do {docker,tar.gz} for all supported platforms
< sipa>
wumpus: good point
< sipa>
do people do docker for more than just x86_64 linux?
< luke-jr>
can we add a file to the .tar.gz to run it as a docker?
< MarcoFalke>
sipa: Also arm
< MarcoFalke>
wumpus: Which is why I'd rather publish a docker script
< wumpus>
i'd be surprised if it wouldn't be used on arm64
< wumpus>
as for the other platforms, probably rarely
< sipa>
MarcoFalke: okay, i'm not all that familiar with the different mechanics
< wumpus>
MarcoFalke: agree, it's the same binaries anyhow packaged differently
< MarcoFalke>
sipa: the docker script builds the docker image, so it is just one step less to do for us
< sipa>
and your suggestion would be to have the script inside the release tgz?
< MarcoFalke>
jup
< sipa>
seems reasonable
< sipa>
(plus having some sort of CI to see if the script works?)
< wumpus>
yes
< MarcoFalke>
hmm, maybe the idea isn't that good because we couldn't reference the gitian built binaries in the script
< luke-jr>
why not?
< wumpus>
as i see it it'd convert a set of gitian-built binaries to a docker image, deterministically
< MarcoFalke>
ok, so the user has to provide the binaries. fair enough
< luke-jr>
the binaries are right there with the script, no?
< sipa>
if it's deterministic to do so even in a fairly uncontrolled environment, that'd still permit publishing the resulting hashes
< * dongcarl>
looks back at `guix pack --format=docker`
< sipa>
dongcarl: o?
< wumpus>
luke-jr: oh... of course
< wumpus>
luke-jr: it'd be shipped with the binary tarball, not the source one
< luke-jr>
right
< sipa>
right
< luke-jr>
(well, part of source too I assume)
< wumpus>
right it has to come from somewhere
< wumpus>
dongcarl: hah!
< * dongcarl>
still trying to find the goal of this discussion
< sipa>
dongcarl: it seems docker is popular (i don't know why, i've never used it), so it'd be great if we have a direct path from our release process to something people can run directly
< wumpus>
yes, for example btcpayserver uses docker IIRC?
< dongcarl>
sipa: Stupid question: why aren't the release binaries enough for this?
< jonatack>
yes iirc dockre is btcpay's recommended installation method
< achow101>
dongcarl: people are just using docker images published by other people that contain the release binaries. afaiu, the point is for us to publish those images instead
< sipa>
dongcarl: that requires people to build their own image from it, using a script they get from... somewhere?
< sipa>
MarcoFalke has a point that just making publishing the script as part of the process may be enough
< wumpus>
it would use the same release binaries, just package them
< sipa>
i think publishing the image directly would be slightly more convenient (and also allow others to build on top of it, i think?)
< achow101>
in the end, it's still our release binaries, but the images themselves may contain other stuff that could be malicious. hence the desire for the developers to create an official one
< MarcoFalke>
sipa: The image can still be published, even if we include only the script in the release
< wumpus>
i guess it's bascially glibc and the bitcoind binary?
< wumpus>
if we'd link bitcoind statically with musl libc there wouldn't even be dependency on that
< luke-jr>
MarcoFalke: what features are even possible?
< MarcoFalke>
"I want to supply the cookie file from outside"
< sipa>
"I want it to mine signet!"
< luke-jr>
that's not runtime?
< MarcoFalke>
"I want to run the gui in docker"
< wumpus>
a docker image with bitcoind that's, just bitcoind :)
< dongcarl>
I'm not entirely sure I see the point in this, let them use the binaries, it's currently clear that the Dockerfiles are not official, and we can't account for all features. Is there a particular event that makes us more worried now?
< sipa>
dongcarl: no
< wumpus>
MarcoFalke: nah
< MarcoFalke>
dongcarl: ppl are asking for it for years
< sipa>
i think the dockerfiles being unofficial is suboptimal
< sipa>
s/official/not going through the same review process/
< wumpus>
there's plenty of ways to run sandboxed GUI (flatpak, snap) there's no point using docker for that, everyone uses docker for server components
< dongcarl>
I think we should do what we do for the systemd services
< dongcarl>
Make it clear that it's just for reference
< dongcarl>
If we do add a Dockerfile I don't want it to bloat over time
< sipa>
yes, that's a delicate line... but i think it'd be better if there was an actual dockerfile that was reviewed and known to be kept working
< sipa>
enough on this i guess
< wumpus>
okay, let's conclude the meeting then, thanks everyone for attending
< sipa>
luke-jr: a comment on your addition to the release notes... i think it's confusing (mixing measuring support/opinions vs measuring safety of updating), and i don't think it means much in any case as update patterns for major versions are very different from minor ones
< dongcarl>
Would love to get some review on #19683, which is blocking Guix progress right now.
< sipa>
that said the release notes should probably say something about non-active taproot support being included?
< luke-jr>
sipa: the goal is to encourage users to mimick the softofrk upgrade pattern
< sipa>
luke-jr: yes, i understand, but i don't think it achieves that
< sipa>
major releases are fundamentally different from minor ones
< luke-jr>
sipa: not perfectly, no, but it should get us closer?
< sipa>
as is, i think the comment adds more confusion than it helps anything
< luke-jr>
right now it looks like upgrades take a very long time
< wumpus>
luke-jr: can you rebase #14066 please? I think it makes sense to add a new platform in the beginning of the release cycle for 0.21, not at the end, or should I pick it up?
< jonasschnelli>
scanning a xpub with scanblocks (#20664) from block 400'000 to tip on a standard machine takes 1m35s (m/0/* m/1/*, range 0-250). That unexpected fast IMO.
< jonasschnelli>
Combining this with rescanblockchain (importing older wallets),..
< jonasschnelli>
and add support for blockfilters in prune mode
< jonasschnelli>
from genesis ist also only 2m13s
< sipa>
jonasschnelli: the scanblocks approach can't be used for pre-descriptor wallets iirc, because we don't have a way to enumerate all sPKs that match a wallet (i think; achow101 may have)
< luke-jr>
sipa: eh, I would think it's easier for older wallets
< jonasschnelli>
sipa: good point
< luke-jr>
oh, because of the key mutability
< achow101>
there's a commit in the descriptor wallet migration pr to compute all spks for a legacy wallet
< sipa>
luke-jr: take things into account like importing a multisig script may result in suddenly making the P2SH version of it watched, but only under the condition that all the related keys are in the wallet etc
< sipa>
it's definitely possible, just nontrivial
< sipa>
achow101: ok cool
< jonasschnelli>
I guess fastscan versus deepscan could be an option for legacy wallets...
< jonasschnelli>
I guess there is no way to detect whether a deepscan is necessary for a legacy wallet before actually completing a deepscan and compare with a fastscan.
< sipa>
not sure what those terms mean
< jonasschnelli>
fastscan (blockfilter), deepscan (how we do it now)
< sipa>
they shouldn't ever differ?
< jonasschnelli>
[...] because we don't have a way to enumerate all sPKs that match a wallet (i think; achow101 may have)
< sipa>
if we have the code for doing the fast one, and we're confident in it, we should use it
< sipa>
otherwise we shouldn't do it at all
< sipa>
jonasschnelli: i should have said "that is blocked on having such functionality, which is nontrivial" - but it's not impossible
< jonasschnelli>
ok
< sipa>
fwiw, i've enabled cirrus CI for the bitcoin-core/secp256k1 repo
< sipa>
is that potentially using up some credits someone paid for?
< sipa>
achow101: i'm confused about 20871... it looks like p3yot3 is already using 0.21 code?
< achow101>
sipa: I'm pretty sure the problem is that the wallet version is 169900 but there is no hd seed
< achow101>
the only way to resolve that is to do the upgrade again with 0.21
< achow101>
this is quite an unfortunate bug. in 0.17, we added upgrading to hd wallets, but forgot about encrypted wallets. if you tried to do that with an encrypted wallet, the version number would be increased, but no hd stuff created
< sipa>
so what's the problem? i thought he didn't want to run pre-release code, but he clearly already is
< achow101>
the problem is he didn't do upgradewallet
< achow101>
I'm writing a response with some more detail