< wumpus>
yanmaani: we limit every network to 512 to limit how much memory and executable size is consumed by this
< bosch-0>
Next Bitcoin Core design call is scheduled for next Wednesday at 9:00AM UTC - https://github.com/BitcoinDesign/Meta/issues/40 - For discussion around this join ##bitcoin-core-gui
< vasild>
jnewbery: isn't that the only one big-endian?
< zkao>
if maxmempool < current mempool, does anyone know what is the criteria for keeping or dropping txs from mempool?
< zkao>
does higher fee txs make lower fee txs drop from mempool?
< wumpus>
vasild: yes, that's the only reason it wasn't disabled months ago, it's kind of unreliable, I guess there's a shortage of that hardware at travis
< wumpus>
it's already been toned down to only compiling and running unit tests iirc
< vasild>
does cirrus provide big endian hw?
< sanket1729>
I see that the uses for functions for CHash256().Write(Span) and CSha256().Write(ptr, len) are inconsistent. Is it worth creating an issue to highlight this?
< sanket1729>
I don't know the difference, or which is better but would be great to have those be consistent
< sanket1729>
It looks like using Span is the better way, I can make a PR for that :)
< fanquake>
wumpus / sipa: the copyright related discussion in #20234 is getting out of control. I'm pretty sure the user "JabbaDesilijicTiure" is just taking the piss at this point. I've deleted their latest comment, but if they continue you could probably just ban them.
< queip>
fanquake: FYI this is same guy that in past days is running around various IRC channels, threatens to kill Bitcoin developers. He rambles about Zeronet (which he ~falsely claims to be top developer of), about politics, bitcoin wiki, uses VPN, switches nicks, runs some scam foundation for "human rights"
< real_or_random>
"we are announcing that travis-ci.org will be officially closed down completely no later than December 31st, 2020, allowing us to focus all our efforts on bringing new features and fixes to travis-ci.com "
< MarcoFalke>
Finally. No travis builds means no travis failures !!11!
< luke-jr>
sigh
< luke-jr>
"Travis CI will continue to offer a free tier for public or open-source repositories on travis-ci.com and will not be affected by the migration."
< luke-jr>
so it sounds like this is just consolidating platforms?
< real_or_random>
I think consolidating is the goal
< real_or_random>
but I don't understand the FAQ entry on write access
< real_or_random>
will they have write access or not?
< luke-jr>
sounds like "yes, but we promise not to use it, and are trying to avoid needing it by getting more granular permissions from GitHub"
< luke-jr>
we probably trust GitHub's security stuff too much anyway4
< real_or_random>
even though this page has instructions for migrating manually, and doesn't mention that they'll do it for us. And they can't because they need the permissions?
< luke-jr>
is there a post-fetch hook to verify the merge signature?
< real_or_random>
I don't understand what's going on. Maybe they're just moving their server so .com, so there are not many ressources for .org left, and that's why the builds ar slow
< real_or_random>
when I try it for a local fork, it says
< real_or_random>
" Read access to code, metadata, and pull requests "
< real_or_random>
" Read and write access to checks, commit statuses, deployments, and repository hooks "
< real_or_random>
I think I understand now... it's a mess
< real_or_random>
The Travis "GitHub App" (which can be installed on repo) does not require read/write access
< wumpus>
heh that sounds overly dramatic for basically just a rename
< real_or_random>
The Travis Github OAuth thing (that you need for logging in on Travis) requires full read/write access to all repos you own.
< real_or_random>
or not. it only says " Full control of private repositories"
< bitcoin-git>
[gui] luke-jr opened pull request #125: GUI: Enable changing the autoprune block space size in intro dialog (master...intro_prune_size) https://github.com/bitcoin-core/gui/pull/125
< bitcoin-git>
[bitcoin] achow101 opened pull request #20262: tests: Skip --descriptor tests if sqlite is not compiled (master...tests-check-sqlite) https://github.com/bitcoin/bitcoin/pull/20262
< luke-jr>
#proposedmeetingtopic allowing sqlite wallet regression into 0.21
< jonasschnelli>
luke-jr: can you elaborate why it is a blocker? What is broken without it?
< luke-jr>
jonasschnelli: without it, sqlite wallets are missing a unique id
< jonasschnelli>
luke-jr: what user function does it break?
< luke-jr>
jonasschnelli: BDB did this for us, but sqlite does not
< MarcoFalke>
why would they need one?
< luke-jr>
jonasschnelli: not having a unique id is a regression from the old wallets
< hebasto>
is ryanofsky here?
< luke-jr>
MarcoFalke: to distinguish between different wallets and renames/moves/copies
< achow101>
the question is what unique ids do for us and whether not having one is a regression
< achow101>
because we don't use them for anything currently
< luke-jr>
achow101: Knots does
< jonasschnelli>
have we expose the unique ID over our APIs or in the GUI?
< achow101>
jonasschnelli: no
< luke-jr>
it's an existing and long standing wallet feature
< luke-jr>
also, we do up until now use them to disallow opening the same wallet twice
< achow101>
it's a "feature" that's a byproduct of bdb and was only used by the bdb handling previously
< sipa>
is it a feature, or a necessary restriction?
< jonasschnelli>
I see luke-jr point... seems like a Knots issue too me. Core has never published or promised unique wallet id... however,.. seems to be easy to fix
< luke-jr>
sipa: the unique id is certainly a feature
< achow101>
I have no strong opinion either way, just that the way the luke-jr originally proposed was not the right way to add an id
< jonasschnelli>
A uuid per wallet seems neat and useful. I agree with luke-jr
< luke-jr>
jonasschnelli: I can't fix it in Knots alone; I would have to simply remove all functionality that uses it
< sipa>
i tend to agree that it's a potentially useful feature
< sipa>
but i don't know if it's a feature right now we need to make promises about
< jnewbery>
The S390x builds on travis are very flakey and often fail, presumably because of contention issues at travis
< luke-jr>
(should we also talk about the travis-ci migration?)
< sipa>
is it just s390x?
< sipa>
all of travis is pretty much broken for the secp256k1 repo
< luke-jr>
:/
< sipa>
s390x was the first sign, but soon after, everything stopped
< jonasschnelli>
yeah.. I personally tend to ignore the CI icons ons pulls... its getting non-useful
< jnewbery>
they were added to test under a big-endian platform. I'm not sure how many bugs have been caught by adding it, but I don't think it's worth keeping it given the costs of having to frequently rerun failed builds
< emzy>
Is someone actually using bitcoin core on x390x?
< sipa>
let me check if that's still the case
< luke-jr>
emzy: it's the only Big Endian platform Travis had
< sipa>
emzy: no, it's just to guarantee big-endian compatibility
< jnewbery>
s390x seems to be the least reliable
< luke-jr>
jnewbery: we could have a bot that restarts s390x if the rest pass?
< luke-jr>
but should we wait and see if .com has this issue?
< sipa>
(which, even if nobody actually uses bitcoin core on a big endian platform, is a good test for the correctness of the code, as it tends to expose implemention-dependency that isn't otherwise exposed)
< MarcoFalke>
s390x found at least one bug, so I think it is useful
< luke-jr>
Travis is only giving us 1 month to move, so..
< jonasschnelli>
CIs are loosing their value if there are a large percentage of false "positives" (false fails)
< sipa>
jonasschnelli: indeed, rapidly
< sipa>
luke-jr: is power9 big endian?
< sipa>
(typically)
< luke-jr>
sipa: it's bi-endian, but Travis only supports LE afaik
< jnewbery>
MarcoFalke: was that a new bug that was introduced, or one that had been around for a long time?
< sipa>
luke-jr: on your power9 system, is the OS LE or BE?
< luke-jr>
sipa: I run LE myself
< MarcoFalke>
jnewbery: both. So it was two bugs actually
< jonasschnelli>
I still recommend to continue build run our own CI system. bitcoinbuilds.org is a great start, ... runs more or less smooths since months,... can be easly extended and is cheeper than travis
< hebasto>
and quicker
< MarcoFalke>
jonasschnelli: We need integration with GitHub, otherwise no one will notice a failing build or even check them
< jonasschnelli>
MarcoFalke: I have it
< jonasschnelli>
Just not on the master branch....
< jonasschnelli>
Will add it to the GUI repo soon
< jonasschnelli>
It's not complicated
< luke-jr>
who runs bitcoinbuilds.org?
< jonasschnelli>
luke-jr: me
< MarcoFalke>
Also, we need to be able to modify the config in-tree, not out-of-tree
< jonasschnelli>
Yes. I'll add that soon.
< sipa>
jonasschnelli: what architecture(s) does it support?
< jonasschnelli>
Not saying it can replace travis,... but it may show the road to a successful and fast CI which we can easly maintain ourselfs
< luke-jr>
can we make it use the GCC compile farm? :P
< jonasschnelli>
sipa: its using libvirt under the hood
< MarcoFalke>
Can it run qemu-s390x?
< luke-jr>
jonasschnelli: that doesn't answer.. :P
< MarcoFalke>
I think it is plain amd64 architecture
< jonasschnelli>
Currently it can only run ubuntu18,... but we could add other servers with other architectures or qemu others (slow)
< luke-jr>
GCC's farm has also ppc64, aarch64, sparc64, and mipsel
< luke-jr>
jonasschnelli: can it work with just a shell?
< jonasschnelli>
luke-jr: how do you mean that? no web frontend?
< luke-jr>
jonasschnelli: I mean SSH into GCC's farm and run builds there
< jonasschnelli>
would be possible...
< jonasschnelli>
contribution welcome
< luke-jr>
another project is already using it for their CI
< sipa>
which project?
< luke-jr>
"HansLambermont uses the compile farm for continuous build integration of the Stellarium project."
< jonasschnelli>
as said... running and expanding our own CI seems to me the best way forward to finally get back to a CI state that is useful
< luke-jr>
"Stellarium is a free open source planetarium for your computer. It shows a realistic sky in 3D, just like what you see with the naked eye, binoculars or a telescope."
< jonasschnelli>
Our demand of funcionability goes beyond most CI product offerings,...
< jnewbery>
I find it hard to believe that there isn't a reliable CI service that we can use and that we need to invent our own
< emzy>
jonasschnelli: I'm open to help with it.
< jonasschnelli>
emzy: great to hear.
< sipa>
jnewbery: indeed...
< jonasschnelli>
jnewbery: me too.. but our demand is non-normal
< sipa>
does jenkins still exist?
< jonasschnelli>
self hosted?
< MarcoFalke>
jnewbery: All CI services are "lol we only support docker"
< sipa>
MarcoFalke: is that a problem?
< MarcoFalke>
sipa: Doesn't jenkins count as running our own?
< sipa>
MarcoFalke: yes
< sipa>
there is a different between running our own and inventing our own :)
< MarcoFalke>
sipa: They don't run sanitizers or wine or ...
< jonasschnelli>
sipa: it's already invented. It runs since 1 year
< jonasschnelli>
(but missed additional features)
< jonasschnelli>
*misses
< sipa>
MarcoFalke: i'm confused what that has to do with docker
< sipa>
sure you can find a docker image that contains sanitizer-enabled compilers and/or wine?
< MarcoFalke>
sipa: You'll have to start the docker daemon with additional permissions
< luke-jr>
so as far as Travis goes, are we just not migrating?
< sipa>
i think we should keep our options open
< MarcoFalke>
sipa: On travis we can start our own docker daemon
< MarcoFalke>
luke-jr: I'd be surprised if travis improved by changing the domain name
< sipa>
MarcoFalke: ok, i probably don't know enough about docker then... i'm not sure what that means or implies
< luke-jr>
MarcoFalke: well, .com has historically been their commercial product?
< sipa>
things could improve with travis.com... i'm not sure
< sipa>
a question is if that requires giving them write permission to the repo, are we open to doing so?
< MarcoFalke>
We used to send them money and support was still ignoring requests
< luke-jr>
sipa: apparently it's write permissions to the user, not the repo specifically :/
< sipa>
luke-jr: we can create a dummy bitcoin-core-ci user or something, no?
< luke-jr>
MarcoFalke: they responded to me (maybe not timely or helpfully, I forget the details)
< luke-jr>
sipa: maybe, but I think that violates GitHub's ToS
< MarcoFalke>
luke-jr: well response is "lol, we don't support that" or "we'll discuss this internally"
< luke-jr>
MarcoFalke: ☺
< MarcoFalke>
luke-jr: Then DrahtBot would violate GitHub's ToS
< jonasschnelli>
Migrations to .com makes sense to me as we have no alternative ready that has the same depths of testing... and it might get better (or worse?)
< jonasschnelli>
back to the topic.... should we disable S390x?
< luke-jr>
MarcoFalke: aha, they have an exception for bots, nm
< MarcoFalke>
jonasschnelli: ACK
< jonasschnelli>
ack
< MarcoFalke>
jonasschnelli: Well, we could move it to cirrus ci for now
< sipa>
if it doesn't work, it should be disabled
< hebasto>
ack on disabling
< luke-jr>
MarcoFalke: Cirrus has BE? :P
< MarcoFalke>
in qemu
< luke-jr>
>_<
< MarcoFalke>
They are working on getting native arm though
< * luke-jr>
should try getting x86_64 gitian working on his ppc64 host via qemu…
< luke-jr>
(apparently I will have to patch qemu)
< sipa>
even just doing BE builds in qemu for master (and not for every PR) would be a major win over not having it at all
< sipa>
same with some other platforms
< MarcoFalke>
I'll create a pull to move it to cirrus
< sipa>
that'd be great
< jonasschnelli>
Indeed. Building **everything** per each PR push is crazy
< jonasschnelli>
(but great if it works)
< jnewbery>
why is it crazy? It seems optimal if it works
< luke-jr>
don't agree on crazy :P
< jonasschnelli>
s/crazy/crazy-system-demanding
< sipa>
if we have the resources, sure
< jonasschnelli>
and it seems like we don't have the resources because large percentages of the builds fail for no reason
< jnewbery>
cpu time is cheaper than developer time
< luke-jr>
fail isn't a resource issue
< luke-jr>
resource shortage would just mean long waits
< sipa>
indeed
< jonasschnelli>
jnewbery: agree... but if you can't be sure whether you have to debug the build or if it was just a glitch in the CI,... its a waste of time
< sipa>
we're not low on resources, it's just broken, and we need to find something that works
< sipa>
whatever we find is better than something that's broken
< luke-jr>
ideally CI would prioritise builds, but otherwise include everything
< MarcoFalke>
I don't think resources are the problem. The problem is the lack of ci infrastructure to trigger builds on working hardware, and someone who maintains the ci infrastructure.
< luke-jr>
eg, put master above PRs
< luke-jr>
maybe have a button where someone can say "I'm actually waiting on this"
< jonasschnelli>
we need a CI-infrastructur-maintainer *duck*
< luke-jr>
jonasschnelli: I thought you just volunteered? :P
< MarcoFalke>
I volunteer jonasschnelli
< jonasschnelli>
no way...
< jonasschnelli>
it needs to be some linux/server crack
< luke-jr>
[19:22:22] <luke-jr> who runs bitcoinbuilds.org? [19:22:26] <jonasschnelli> luke-jr: me
< luke-jr>
we'll say that counts
< jnewbery>
MarcoFalke: I agree. The problem is that we don't have a build manager
< jonasschnelli>
I start stuff,... but not finish them
< luke-jr>
XD
< sipa>
jonasschnelli: i fear this will be a problem if we'd migrate to bitcoinbuilds.org for part or all of our CI
< sipa>
things will break, and will require maintenance
< wumpus>
oh crap don't tell me i missed the meeting almost because of the DST change here *facepalm*
< jonasschnelli>
sipa: we should certenly not migrate before a group of people have commited time to it
< luke-jr>
wumpus: lol
< sipa>
wumpus: still 17 minutes left!
< jnewbery>
I think all these problems are solvable by someone who has the time and desire to figure out the problems. Part of that might be researching different CI options
< MarcoFalke>
wumpus: lol
< jonasschnelli>
wumpus: hah
< sipa>
wumpus: DST is evil
< wumpus>
sipa: yess
< luke-jr>
wumpus: nobody will ever know: the bot is broken to
< luke-jr>
too*
< jonasschnelli>
Ideally we have a team of people willing to maintain the CI
< MarcoFalke>
jnewbery: I did look at all available ci options a year ago
< jonasschnelli>
If it is all on my shoulders,.. it will break for sure
< luke-jr>
jonasschnelli: like we don't have our hands full maintaining Core
< MarcoFalke>
As I said most of them are docker-only, so they can't run our scripts
< dongcarl>
If we have actual needs beyond what CI providers offer, then we should pay someone to maintain a CI cluster, no?
< jonasschnelli>
luke-jr: there are enought people with server admin skills but no coding skills willing to contribute.
< luke-jr>
dongcarl: that implies we have excess funding
< MarcoFalke>
Only cirrus ci and travis offer full vms
< jonasschnelli>
dongcarl: sure. Paing for it should be possible
< sipa>
funding can be found, i'm sure
< jnewbery>
I expect funding will not be a problem
< luke-jr>
oh, so it's only my funding that's a problem :/
< luke-jr>
sigh
< yanmaani>
luke-jr: Are build servers *that* expensive?
< jonasschnelli>
no... but the maintenance men.hours
< luke-jr>
on that note, funding might be good for security bug bounties too
< luke-jr>
yanmaani: no, but maintaining a CI platform probably is
< yanmaani>
oh right, misread
< emzy>
I think a CI team is better than a single persion.
< jnewbery>
*or woman hours :)
< jonasschnelli>
emzy: for sure
< jonasschnelli>
jnewbery: oops. yes.
< MarcoFalke>
Have people had issues with Cirrus Ci lately?
< luke-jr>
MarcoFalke: I started ignoring it because when it does have problems I can't restart it
< sipa>
MarcoFalke: i can't remember any... i also have no idea what actually runs on it
< MarcoFalke>
They used to abort sometimes, but setting kvm:true fixed it I think
< MarcoFalke>
sipa: Only the thread sanitizer and one fuzzer
< sipa>
is there any reason why everything couldn't run on cirrus?
< MarcoFalke>
We could also move all jobs to Cirrus Ci, which would probably mean we need to pay them
< MarcoFalke>
free tier is only 8 cpus or so
< luke-jr>
probably cheaper than maintaining our own
< luke-jr>
if it works decently
< MarcoFalke>
probably
< dongcarl>
Do we have any needs beyond what Cirrus provides?
< MarcoFalke>
dongcarl: s390x and arm would be nice, but we can use qemu
< luke-jr>
jonasschnelli: but that's just Docker?
< MarcoFalke>
jonasschnelli: If it is just docker, it won't work
< jonasschnelli>
luke-jr: they have x86 and macOS... all docker I guess
< MarcoFalke>
If someone knows another non-docker ci, let me know, but I don't think there is
< luke-jr>
MarcoFalke: hmm, does KVM work inside stock Docker?
< bitcoin-git>
[bitcoin] gwillen opened pull request #20264: test: Make secp tests optional in `make check` (master...feature-optional-secp-check) https://github.com/bitcoin/bitcoin/pull/20264
< MarcoFalke>
luke-jr: we don't use kvm
< MarcoFalke>
I doubt it does
< MarcoFalke>
qemu-user works in docker
< luke-jr>
MarcoFalke: we could
< MarcoFalke>
luke-jr: Patches welcome :)
< MarcoFalke>
I don't have background in kvm, so I can't help here
< luke-jr>
MarcoFalke: without a plan, no point
< luke-jr>
MarcoFalke: KVM is just virtualising a machine - so you can do anything as if it was real hardware
< MarcoFalke>
luke-jr: git grep docker ./ci
< MarcoFalke>
then replace docker with kvm
< luke-jr>
sure, but unless Docker supports KVM, it won't help
< MarcoFalke>
jup, I don't know an answer to that
< jonasschnelli>
Other topics?
< jonasschnelli>
(short ones)
< jonasschnelli>
#endmeeting
< jonasschnelli>
\o
< jnewbery>
thanks jonasschnelli!
< luke-jr>
wumpus: okay, you can start your meeting now :P
< jonasschnelli>
hehe
< wumpus>
yes, thanks for picking it up jonasschnelli
< dongcarl>
MarcoFalke: can we do s390x qemu on Travis x86? Or way too slow for that?
< MarcoFalke>
dongcarl: we can. It should only be a bit slower
< wumpus>
it would be more efficient to do the build using cross-compile, then run only the unit tests using qemu
< MarcoFalke>
wumpus: That is what it does
< MarcoFalke>
So compilation is the same cost
< wumpus>
ok, didn't realize that,I thought it did the whole build on s390s
< sipa>
i thought it was actually running on s390x hardware
< wumpus>
qemu-user is *weird* with differnt endianness though
< MarcoFalke>
sipa: Right now it does (well if it worked)
< sipa>
MarcoFalke: oh ok
< sipa>
so what is the "That is what it does" about?
< emzy>
Is qemu acurate enough to really find problems with BE?
< MarcoFalke>
Though, we can toggle one line in the config to use qemu-user
< sipa>
emzy: definitely
< wumpus>
qemu system emulation is
< * dongcarl>
wishes Travis had binfmt_misc qemu images
< wumpus>
qemu user emulation is *not* for swapped endianness
< MarcoFalke>
sipa: If the ci system were using qemu, it would cross compile and only run qemu-user
< MarcoFalke>
wumpus: Oh, it is not? I didn't know that
< wumpus>
it fails to swap some linux kernel structures
< wumpus>
maybe it's better today but there used to be quite some problems making the platform much stranger than an actual big-endian machine :-)
< MarcoFalke>
I guess the question is: will qemu-user be able to find endian issues in the Bitcoin Core codebase
< MarcoFalke>
We don't care about issues in the linux kernel
< wumpus>
it found some false positives at the time
< luke-jr>
qemu-system has trouble with x86 on ppc due to the memory model differences :x
< wumpus>
no, but it communicates with the linux kernel
< sipa>
this should be easy to figure out
< sipa>
if qemu-user works with s390x
< wumpus>
every field in every structure that's directly or indirectly exchanged needs to be endian-corrected
< wumpus>
that's not a trivial task and error prone
< luke-jr>
whatever CI we go with, couldn't we just buy them a POWER system and run it in BE? :P
< luke-jr>
rather than trying to sort out emulation mess
< wumpus>
which is why I doubt whether it's a good reflection of actual hardware a bit
< MarcoFalke>
luke-jr: They use google cloud and packet or amazon. Good luck buying them a POWER system
< luke-jr>
(actually, POWER8/9 should be capable of running VMs in both endians..)
< luke-jr>
MarcoFalke: ugh
< jonatack>
wumpus: arf i missed it too due to DST * makes note
< wumpus>
jonatack: hah better next week
< jonatack>
hehe
< emzy>
wumpus online calendar worked perfectly for me. :)
< wumpus>
emzy: glad to hear that :-)
< luke-jr>
KOrganizer 4.4 here <.<
< emzy>
Apple Calendar here <.<
< wumpus>
in any case I don't have anything against a qemu-user based solution for BE if it actually works now and doesn't require weird workarounds in the tests
< MarcoFalke>
wumpus: I'll test :)
< wumpus>
MarcoFalke: thanks!
< sipa>
let's see how far i get with qemu-user s390x in a debian vm
< bitcoin-git>
[bitcoin] mjdietzx opened pull request #20265: refactor: get wallet path relative to wallet_dir (master...refactor-get-relative-wallet-path) https://github.com/bitcoin/bitcoin/pull/20265
< sipa>
segfault when running a hello world :(
< sipa>
it works with aarch64
< bitcoin-git>
[bitcoin] mjdietzx closed pull request #20265: refactor: get wallet path relative to wallet_dir (master...refactor-get-relative-wallet-path) https://github.com/bitcoin/bitcoin/pull/20265
< wumpus>
good to know, yes i'd expect aarch64 on amd64 to be very well tested
< jonasschnelli>
I'm even failing setting up debian stretch s390x with qemu-system-s390x...
< sipa>
going to try with testing
< sipa>
wumpus: hello world works in debian testing!
< real_or_random>
being able to run things on qemu is certainly useful but I think the bigger issue is the travis migration
< sipa>
real_or_random: trying to get that to work locally
< sipa>
running into weird zfs issues
< real_or_random>
what kind of issues?
< sipa>
apparently the ubuntu docker install creates zfs filesystems for each docker image or something, unsure
< sipa>
and now there is one in use which it can't destroy
< sipa>
which i don't understand as it's not mounted
< real_or_random>
sipa: which command are you trying to run?
< sipa>
real_or_random: the first ons, run --rm -t -v ...
< real_or_random>
hm, yeah, no idea. that one run on successfully also on travis.
< sipa>
real_or_random: ok, works
< sipa>
i had to remove the "--rm"
< bitcoin-git>
[bitcoin] achow101 opened pull request #20267: Disable and fix tests for when BDB is not compiled (master...tests-opt-sqlite-bdb) https://github.com/bitcoin/bitcoin/pull/20267