< vasild>
kallewoof: I am going to test https://github.com/bitcoin/bitcoin/pull/17994 today and I wonder if there is a better way than just start it to download blocks and try to observe how it behaves. Is there a way to test in a controlled environment where we feed out-of-order blocks to it deterministically?
< kallewoof>
vasild: nice! you'll get out of order blocks for sure just by running it for a bit, is my experience. One thing you can do is put in some LogPrintfs when it hits the "finalize" case in case you wanna check that.
< provoostenator>
vasild: or you could write a regest for that, although we'd need more fine grained control of the regtest nodes p2p behavior
< provoostenator>
Not sure how far you can get with submitblock RPC.
< vasild>
provoostenator: I found out that https://github.com/bitcoin/bitcoin/blob/master/src/test/validation_block_tests.cpp#L151 is very close to what's needed to reproduce the out-of-order block feeding wrt #17994. I am now replacing its random multi-threaded call of ProcessNewBlock() with a signle-threaded deterministic out-of-order block supply. Lets see what happens...
< vasild>
at least I will be able to observe the issue and its fix manually from the log, not sure if it will be possible to completely automate and add as a test. Is it possible to check if something was or was not printed in the log from a BOOST_AUTO_TEST_CASE()?
< jonatack>
vasild: I don't know the unit tests suite well yet, and there may be better examples, but maybe have a look in
< wumpus>
pinheadmz: no, no idea, I do know the list of testnet seed nodes is hardly ever updated, seems the last time was 2015 (and that was mostly a move) so it's likely hugely out of date too
< wumpus>
it probably doesn't make sense, e.g. wasteful, to hardcode a long list like we do for mainnet, but it would be good if someone checked the testnet list for staleness
< emilengler>
hebasto: Will review this PR tomorrow
< provoostenator>
^ I'll try to conserve ACKs but can quickly address feedback if needed on
< wumpus>
that would be really nice to have
< wumpus>
achow101: is 18204 a feature or performance improvement?
< wumpus>
(I mean if the latter it could potentially go in after the feature freeze)
< achow101>
it can go in after the feature freeze I guess
< achow101>
it's really a performance improvement
< sipa>
i think it's a weak performance improvement, but also a necessity for descriptor wallets
< wumpus>
but it has quite some ACKs, I see, so might not need to wait that long anyhow
< achow101>
I just don't want it to get stuck for another few months with 2 acks
< wumpus>
it's always ok to remind me if something is almost ready for merge btw no need to wait until the meeting
< wumpus>
#topic PPA URI (luke-jr)
< sipa>
pinging BlueMatt
< sipa>
ah, he's going through airport security
< cncr04s>
remove that 3 second wait on the send button
< sipa>
?
< cncr04s>
or at least add an option
< wumpus>
I'll let BlueMatt and luke-jr fight this out
< MarcoFalke>
I think the only question was whether to use deterministic builds or not
< MarcoFalke>
I can't see an argument for non-deterministic builds
< sipa>
and whether we want to support PPAs at all
< wumpus>
I think that's the main question
< achow101>
there's various documentation written by other people that refer to the ppa
< sipa>
if we can make the PPA deterministic (or even identical to the release builds), that would be ideal
< wumpus>
yes, definitely
< luke-jr>
I am maintaining the PPA
< sipa>
<BlueMatt> Note that ppa has two series’ of issues that releases did not: GUI issues that persisted for two releases and weren’t ever solved and the 32-bit test failures.
< luke-jr>
I don't really know why anyone else needs to be involved in this fact..
< sipa>
<BlueMatt> The second one especially scares me
< wumpus>
what do you need from us then?
< luke-jr>
the only question in my mind is whether it should remain at 'bitcoin' as some have suggested, or move to my own launchpad
< MarcoFalke>
That was the gcc compiler bug
< sipa>
<BlueMatt> But if people want it, it should be maintained as a part of the packaging repo
< achow101>
I think it should remain at 'bitcoin' as that's where a bunch of docs that mention the ppa point to
< luke-jr>
BlueMatt seems to not only want to stop maitnaining it, but also suppress others from doing so. hence the meeting topic
< MarcoFalke>
I think we shouldn't offer software that is impossible to audit
< MarcoFalke>
determinisitic builds in the ppa are fine, though
< wumpus>
well the "bitcoin" PPA is his so as for maintining that that's his decision, if you maintain youre own somewhere else that's fine, but we likely won't link to it
< wumpus>
(we never even linked from the repo to his ppa either afaik)
< sipa>
bitcoincore
< achow101>
wumpus: the ppa has been linked to before from both our repo and bitcoincore.org. those were removed, but it has been "official"
< sipa>
bitcoincore.org or bitcoin.org linked to it as a way of installing
< MarcoFalke>
Yes, I remember removing that link
< wumpus>
achow101: I think the ppa was only linked as a means to install bdb4
< wumpus>
oh okay
< MarcoFalke>
I think there is no one objecting a ppa that wraps our normal release builds
< wumpus>
like the snap does, right?
< MarcoFalke>
jup
< MarcoFalke>
But some people don't like the snap, because it doesn't install it in the "classic" location etc
< luke-jr>
… how much of that made it :/
< luke-jr>
Canonical is ultimately responsible for the PPA builds
< sipa>
it's also not what you'd get by building from source and knstalling
< luke-jr>
the gitian builds are terrible; they have a purpose, sure, but they're not even close to what users ideally would use
< wumpus>
luke-jr: your last message was "BlueMatt seems to not only want to stop maitnaining it, but also suppress others from doing so. hence the meeting topic"
< luke-jr>
the PPA is built by the OS vendor, and produces binaries specifically for the OS
< luke-jr>
wumpus: after that was [19:16:12] <luke-jr> so I guess the questions are 1) do we want to keep the PPA at the old URI, and 2) can we satisfy BlueMatt to allow that?
< wumpus>
make the gitian builds less horrible then
< luke-jr>
wumpus: that's incompatible with the goal of them
< wumpus>
it's what everything is based on and the only auditable one
< luke-jr>
wumpus: gitian builds are intended to run anywhere, but that's incompatible with being targetted to a specific distro
< achow101>
luke-jr: how so? something that runs anywhere should also run on a specific distro
< * dongcarl>
is so confused
< MarcoFalke>
agree with achow101
< sipa>
luke-jr: just because of UI theming etc?
< luke-jr>
achow101: ideally, binaries should dynamic link to system libraries for ~everything
< wumpus>
things like GUI costomization/integration I guess
< luke-jr>
sipa: that's a symptom
< wumpus>
not that that ever worked well for the PPA
< wumpus>
we had more UI issues with the PPA than ever with the gitian builds
< sipa>
seema like a small price to pay for auditabke builds
< achow101>
luke-jr: but we don't necessarily support the specific system libraries that may be installed
< sipa>
sorry, car tyoing
< luke-jr>
nobody is suggesting removing the gitian option
< achow101>
there may be version differences, etc.
< luke-jr>
achow101: we do
< wumpus>
although at least the crazyness with ubuntu unity is gone now
< luke-jr>
achow101: the preferred install is from source
< luke-jr>
achow101: (and PPAs don't support distros without the required versions)
< luke-jr>
logical order of preference, for an Ubuntu user, is build-from-source > PPA > gitian
< wumpus>
but I agree with sipa, deterministic builds are worth a little bit of GUI integration annoyance
< MarcoFalke>
If someone really wants to use the system libs, why can't they build from source?
< luke-jr>
it might be nice if we could deterministically make the PPA debs, but Launchpad doesn't support that
< achow101>
if a distro version lacks the requisite system libs, gitian would still work there, no? I think that's a good thing
< luke-jr>
MarcoFalke: many users don't know how, or don't want to spend the time
< dongcarl>
Very naive thought: is it possible to have 2 PPAs, 1 for gitian built binaries, 1 for specifically OS-integrated?
< luke-jr>
achow101: absolutely
< MarcoFalke>
dongcarl: I'd support that
< luke-jr>
achow101: PPAs are not a replacement for gitian, they are an alternative for certain users
< luke-jr>
dongcarl: should be
< MarcoFalke>
bitcoin/bitcoin would be deterministic and luke-jr/bitcoin is built with system libs
< luke-jr>
dongcarl: sounds like a good idea, even
< luke-jr>
MarcoFalke: that seems backward
< luke-jr>
bitcoin/bitcoin has always been system libs
< sipa>
luke-jr: you seem to be the only one arguing for system libs
< luke-jr>
sipa: so?
< achow101>
I think anything "official" should only be determinisitic
< MarcoFalke>
agree
< sipa>
agree
< wumpus>
achow101: +1
< dongcarl>
agree
< fanquake>
+1
< luke-jr>
achow101: why?
< luke-jr>
Distro-built is equivalent security
< luke-jr>
if your distro is compromised, a gitian build won't help you
< wumpus>
because it's the only one we can vouch for based on the sha256 hashes
< MarcoFalke>
luke-jr: We are talking about the ppa infrastructure being compromised, not the normal package build infra
< luke-jr>
MarcoFalke: isn't it the same?
< MarcoFalke>
I hope for Ubuntu that they are different, at least different datacenters
< sipa>
MarcoFalke: i doubt that
< wumpus>
so I think we've pretty much reached an agreement here, any other topics?
< luke-jr>
anyway, how about adding a disclaimer to the effect of "These are built by Canonical, not the Bitcoin Core project"?
< achow101>
luke-jr: how is it the same?
< luke-jr>
achow101: PPAs are built by Canonical
< sipa>
luke-jr: and maintained by the PPA maintainer
< wumpus>
yes but their build infrastructure runs arbitrary builds of arbitrary software, in the PPA case, so it's not that far fetched it could be compromised
< luke-jr>
sipa: just like the snaps are..
< MarcoFalke>
luke-jr: The snap you can check against the signed hash
< luke-jr>
MarcoFalke: our website says you can't last I checked
< wumpus>
right, the snap packages the gitian-built executables so you can verify them in the same way
< MarcoFalke>
luke-jr: I coldn't find a cross-platform way to do it
< MarcoFalke>
But on my machine it works last time I tried
< wumpus>
this would also be true for a PPA that packages the gitian-built binaries
< jonatack>
wumpus: I am not sure the blockers are a topic this week, but FWIW it looks like #16426 is currently replaced by #17954
< luke-jr>
MarcoFalke: it verifies the entire snap, not just the chosen binaries installed?
< jonatack>
which, by mutual agreement of the PR authors, apparently should be merged in first
< wumpus>
jonatack: I've forgone blockers because of focusing on things that need to go in before the feature freeze which is imminent, but sure I'll swap them
< luke-jr>
I'm not entirely sure what argument sipa is trying to make..
< MarcoFalke>
luke-jr: Of course you'd still have to trust the snapd
< luke-jr>
there is no way I could as maintainer compromise the PPA without it being publicly visible
< achow101>
but it's possible for canonical to compromise it invisibly
< luke-jr>
achow101: absolutely.
< luke-jr>
just like they can compromise the OS
< luke-jr>
in which case gitian does no good to prevent it
< achow101>
but at least users can verify that the ppa was not compromised with gitian
< hebasto>
are any estimation of a ppa share among all users?
< achow101>
and IIRC, PPAs can still be used on ubuntu derivatives
< achow101>
and other distros which are not necessarily canonical
< luke-jr>
hebasto: very few have noticed the URI changed
< sipa>
luke-jr: it didn't cba ge
< sipa>
it was discontinued
< luke-jr>
sipa: no, I am still maintaining it
< sipa>
you have your own PPA
< luke-jr>
hebasto: 2019-09, there were still around 8000 users of the PPA at bitcoin/bitcoin
< wumpus>
so we already discussed a compromise acceptable with most people here (two PPAs), I'm not sure it makes sense to continue arguing this
< luke-jr>
who are now stuck on 0.18.0 until they switch to the new PPA
< luke-jr>
wumpus: anyone can make a 2nd PPA, but I'm not sure there's a need with the snap doing the gitian binaries already
< sipa>
maybe we should, until we resolve this, push an update to the PPA that installs a binary that just prints "this is not maintained, see page X"
< luke-jr>
sipa: or at least deletes the binary
< sipa>
right
< sipa>
i agree there are probably people stuck at 0.18 by not noticing the ppa page that says it's not maintained
< luke-jr>
maybe less invasive to patch 0.18.0 with a message, but.. not sure I like the idea of doing a known-vulnerable "release"
< wumpus>
at least make sure it doesn't delete the wallet ...
< sipa>
wumpus: i don't think system installs can delete user files
< wumpus>
sipa: I think that was a risk wit hthe snap at some point
< luke-jr>
I guess that's one potentially scary thing about Snaps
< luke-jr>
yeah
< sipa>
wumpus: i know nothing about snap
< luke-jr>
sipa: it's basically a chroot AIUI
< wumpus>
but yes unintalling a deb shouldn't remove user files (or even system configuration files without --purge)
< MarcoFalke>
Snap creates a snapshot of your wallet on uninstall
< sipa>
MarcoFalke: scary
< luke-jr>
I suppose that might be a reason to support a second gitian-binary PPA
< MarcoFalke>
sipa: Less scary than deleting it
< sipa>
fair
< luke-jr>
back to the original topic though, bitcoin/bitcoin seems like a bad URI IMO
< luke-jr>
bumping it with a move message seems like a good solution
< luke-jr>
and that can refer to the two new PPAs with clarification of distinction
< luke-jr>
?
< luke-jr>
could be luke-jr/bitcoincore & luke-jr/bitcoincore-deterministic, or bitcoincore/bitcoincore-{system,deterministic} or something along those lines?
< luke-jr>
(not promising I'll make the gitian binary PPA - just throwing out ideas for discussion)
< luke-jr>
advantage of the former is that it's obvious who maintains it; but the latter will work even if multiple people or maintainers change
< achow101>
I think we have to keep bitcoin/bitcoin just to keep existing docs working and not confusing existing users further
< luke-jr>
downside of the latter is it implies the project is responsible for it, which seems undesirable
< wumpus>
if there's two PPAs then docs have to be updated anyway
< luke-jr>
achow101: currently only BlueMatt has a monopoly on the 'bitcoin' name
< wumpus>
WITH documentation on what the choice is and why
< luke-jr>
even providing a notice-bump on the old PPA will require BlueMatt's cooperation
< MarcoFalke>
In launchpad it is possible to change the email address to something@bitcoincore.org, so that whoever has access to that can reset it
< MarcoFalke>
That is how I set up the snap
< luke-jr>
MarcoFalke: I don't know how BlueMatt setup the Launchpad stuff - I suspect the account is his personal account, and ~bitcoin is just a team with only him
< luke-jr>
would be nice if we could come to some agreement here to present to BlueMatt.. maybe "two new PPAs, and bump bitcoin/bitcoin with a notice"?
< luke-jr>
actually, notice should mention the gitian builds and Snap too for completeness IMO
< wumpus>
yes +1 with adding a notice to bitcoin/bitcoin at least, no matter if there's any new PPAs
< luke-jr>
it would be IMO absurd to say that BlueMatt is allowed to maintain a PPA and I am not
< luke-jr>
(which is what would be implied by refusing to tell users of the new URI)
< wumpus>
well we all think an 'official' PPA should be built from the gitian-built binaries, and you disagree with that, so that's not entirely unexpected
< luke-jr>
wumpus: it never has been
< ryanofsky>
does PPA require a single maintainer? with snaps we have a github packaging repository that gets normal review and a something@bitcoincore.org owner like marco mentioned
< luke-jr>
ryanofsky: the PPA stuff is in the same repo
< luke-jr>
ryanofsky: I have a PR open for the gitian YML that submits it to Canonical, but nobody seem to care to review it
< wumpus>
that's another point to simply package the gitain-built binaries; it doens't require as much maintenance or separate testing
< ryanofsky>
oh, well it seems kind of important to get that merged so we can have multiple maintainers
< MarcoFalke>
luke-jr: It seems to be pending on the further steps we take
< luke-jr>
ryanofsky: I wasn't aware anyone else was interested ;)
< luke-jr>
MarcoFalke: ultimately, it's a question of whether someone is trying to dictate to users how they use Core, or let them make an informed decision
< luke-jr>
it'd be one thing if nobody was willing to maintain the PPA at all; it's another to try to stop someone
< wumpus>
well you're maintaining it, we're definitely not able to stop you doing that
< luke-jr>
this is the same way it's always been, and the way these users are used to it
< hebasto>
luke-jr: what is "AIUI" you referred to? (19:40:39 UTC)
< luke-jr>
As I Understand It
< hebasto>
thanks ;)
< luke-jr>
wumpus: deceiving users into thinking they must switch to gitian-derived binaries that aren't tailored to their OS, is the opposite extreme from 'official' recognition
< wumpus>
it's the way things were done but we realized that it was not a good idea
< wumpus>
e.g. due to unique bugs in the ppa build
< luke-jr>
so because you think it isn't a good idea, everyone who disagrees is censored?
< wumpus>
what, do you feel censored?
< luke-jr>
wumpus: if there is an intentional effort to deceive users into not knowing they can continue to use a PPA, that seems to fit
< wumpus>
you were allowed to monopolize the entire meeting
< wumpus>
I don't think we could give you a bigger platform for your opinions if we wanted
< wumpus>
that doesn't mean we have to agree with it
< luke-jr>
meetings are between devs; I doubt many users read it
< luke-jr>
well, many probably do, but many do not
< * luke-jr>
wonders if plotting node versions over time would reveal much about how many are 'stuck' on 0.18.0
< hebasto>
ryanofsky: mind looking into #17813 discussion about default configure options? Will appreciate your opinion.
< ryanofsky>
i looked briefly, it seemed like what carl suggested there was the obvious thing to do, not sure if it is difficult to implement. can comment though
< hebasto>
ryanofsky: thanks!
< hebasto>
ryanofsky: do you think the help string is clear?
< ryanofsky>
current one from issue description seems unclear in the way you suggested
< hebasto>
ok
< BlueMatt>
luke-jr: hey, sorry I missed the meeting, it totally didnt cross my mind that I'd be going through tsa during it :(
< BlueMatt>
sounds like the conclusion is "needs to be gitian built, cause no one disagrees with that, but everyone feels somewhat uncomfortable with it being anything else"
< BlueMatt>
which I'm def happy with (been saying it for literally years) - are you interested in rewriting the build scripts luke-jr or is someone else gonna?
< luke-jr>
BlueMatt: I disagree with only gitian static binaries
< luke-jr>
the conclusion was both options
< luke-jr>
and changing bitcoin/bitcoin to a notice of some sort explaining the upgrade paths
< luke-jr>
are you okay with that?
< BlueMatt>
luke-jr: the thing already says its unmaintained. I'll clarify to point out that it is not official and that those who want something more officially maintained they should followup on the packaging issue.
< BlueMatt>
note that I've been complaining about the ppa for years, so this should be a surprise to no one.
< BlueMatt>
and have been indicating that it needs to be gitian-binaries for years
< luke-jr>
BlueMatt: PPA users will probably never even look at that page
< sipa>
BlueMatt: i think one issue that isn't addressed is that plenty of existing PPA installs will never see that page
< sipa>
so an idea would be to push an update that removes the binary at least, or even better, one that prints a notice
< BlueMatt>
right, I dunno what to do there? I can delete the packages but anyone on recentish ubuntu releases will fail to install anyway
< luke-jr>
BlueMatt: it absolutely should not be gitian binaries. That's fine as an option, but it is not a replacemnet.
< luke-jr>
BlueMatt: the idea was to version bump with a text file or something
< luke-jr>
(possibly deleting the installed binaries, which may be controversial?)
< BlueMatt>
ubuntu 19.X will fail to install already, as well as 20.04
< BlueMatt>
let me also delete the packages
< luke-jr>
deleting them won't uninstall them
< BlueMatt>
yes, but an os upgrde will
< BlueMatt>
it warns you on upgrade and requires confirmation to continue
< luke-jr>
OS upgrade probably already dropped them
< luke-jr>
the issue is the ~6000 existing installs who haven't upgraded
< luke-jr>
using LTS
< BlueMatt>
almost all of those distros are unmaintained at the os level?
< sipa>
xenial is still maintained until april 2021
< BlueMatt>
only xenaial and bionic are not
< luke-jr>
even trusty is still maintained IIRC..
< sipa>
trusty is in "extended security maintenance"
< luke-jr>
trusty says EOL April 2022
< luke-jr>
precise is extended security maint
< BlueMatt>
precise is eol for "extendeds ecurity" in 2019
< luke-jr>
wait no, precise is dead, but the wiki is outdated
< sipa>
ESM requires registering with Ubuntu
< luke-jr>
in any case, there are still people using 0.18.0, and likely from matt's ppa
< luke-jr>
they shouldn't be told "switch to gitian binaries or else"; they should be made aware they can continue using system builds via the new PPA
< sipa>
i suspect that some are
< sipa>
luke-jr: if they trust you
< achow101>
luke-jr: can you succinctly say why system libs > gitian? I highly suspect that users of the PPA won't know or really care.
< luke-jr>
sipa: if they trust Canonical; again, there is nothing I can do without it being publicly visible.. but sure, a reasonable notice is fine
< luke-jr>
achow101: system libs integrates better, is more RAM-efficient, and gets bugfix updates (incl security) immediately
< luke-jr>
achow101: I don't know which Ubuntu version, but one has serious usability issues with gitian builds
< BlueMatt>
note that the ppa was *worse* in that regard
< luke-jr>
BlueMatt: ?
< BlueMatt>
though the gui issues are a bit better due to ubuntu dropping their own crap and going back to gnome
< luke-jr>
only got stats for cosmic before BlueMatt deleted the pkgs, but people are still installing it since 2019-09 :x
< BlueMatt>
we cant help anyone with an outdated unmaintained ubuntu distro
< BlueMatt>
launchpad doesnt even let you upload replacement dummy packages
< luke-jr>
sure, but we can help people using maintained versions
< ysangkok>
luke-jr: how big are the RAM savings? wouldn't it only be a few megabytes at most? very small compared to the total RAM usage
< BlueMatt>
thats basically just bionic, which should go away mostly very soon
< luke-jr>
ysangkok: all of Qt?
< BlueMatt>
ysangkok: for reference, most "modern, hip" stuff is moving to static-linking-by-default (rust, go, i mean shit electron apps ship an entire copy of chromium for each application)
< luke-jr>
BlueMatt: which is why I advise nobody use them
< BlueMatt>
but, for us, its super trivial, and allows for testing and maintinence focus, which is critical
< ysangkok>
luke-jr: bitcoin doesn't use "all of Qt". for example, my libqt5core is only 6 MB. and given that the default desktop isn't qt-based, qt may be loaded solely for bitcoin in case it is dynamically linked
< luke-jr>
ysangkok: worst case scenario, it's the same as if it was static linked, yes; but best case (and likely) scenario is better
< sipa>
if 6 MB is a problem, you probably shouldn't be running bitcoin core
< luke-jr>
heh, true
< sipa>
i agree in principle - there are advantages to dynamic libraries
< sipa>
but i don't see how they weigh up against the disadvantages in this case
< luke-jr>
one or two of the distros have a page explaining why it's bad to static link
< luke-jr>
sipa: what disadvantage?
< sipa>
luke-jr: reproducible binaries that can be compared with everyone
< sipa>
being the major one
< luke-jr>
there's no practical advantage to that when comparing to OS vendor built
< luke-jr>
unless the OS itself is deterministic, which none are yet
< luke-jr>
hmm, Debian claims static "renders some security measure less effective (ASLR for example)."
< luke-jr>
is that actually true?
< sipa>
i don't know
< luke-jr>
I would think PIE would at least get us *some* level of ASLR