< wumpus> yanmaani: we limit every network to 512 to limit how much memory and executable size is consumed by this
< bosch-0> Next Bitcoin Core design call is scheduled for next Wednesday at 9:00AM UTC - https://github.com/BitcoinDesign/Meta/issues/40 - For discussion around this join ##bitcoin-core-gui
< vasild> jonatack: so it fails for you too!
< vasild> Precious!
< jnewbery> #proposedmeetingtopic Disable S390x travis build
< vasild> jnewbery: isn't that the only one big-endian?
< zkao> if maxmempool < current mempool, does anyone know what is the criteria for keeping or dropping txs from mempool?
< zkao> does higher fee txs make lower fee txs drop from mempool?
< wumpus> vasild: yes, that's the only reason it wasn't disabled months ago, it's kind of unreliable, I guess there's a shortage of that hardware at travis
< wumpus> it's already been toned down to only compiling and running unit tests iirc
< vasild> does cirrus provide big endian hw?
< sanket1729> I see that the uses for functions for CHash256().Write(Span) and CSha256().Write(ptr, len) are inconsistent. Is it worth creating an issue to highlight this?
< sanket1729> I don't know the difference, or which is better but would be great to have those be consistent
< sanket1729> It looks like using Span is the better way, I can make a PR for that :)
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/3f512f3d5639...6196cf77e574
< bitcoin-git> bitcoin/master d419fde Troy Giorshev: [net processing] Don't add AlreadyHave txs to recentRejects
< bitcoin-git> bitcoin/master 6196cf7 Wladimir J. van der Laan: Merge #19753: p2p: don't add AlreadyHave transactions to recentRejects
< bitcoin-git> [bitcoin] laanwj merged pull request #19753: p2p: don't add AlreadyHave transactions to recentRejects (master...2020-08-clean-tx-processing) https://github.com/bitcoin/bitcoin/pull/19753
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/6196cf77e574...924a4ff7eb62
< bitcoin-git> bitcoin/master fa56d56 MarcoFalke: fuzz: Properly initialize PrecomputedTransactionData
< bitcoin-git> bitcoin/master 924a4ff Wladimir J. van der Laan: Merge #20242: fuzz: Properly initialize PrecomputedTransactionData
< bitcoin-git> [bitcoin] laanwj merged pull request #20242: fuzz: Properly initialize PrecomputedTransactionData (master...2010-fuzzInit) https://github.com/bitcoin/bitcoin/pull/20242
< fanquake> sanket1729: sure
< fanquake> wumpus / sipa: the copyright related discussion in #20234 is getting out of control. I'm pretty sure the user "JabbaDesilijicTiure" is just taking the piss at this point. I've deleted their latest comment, but if they continue you could probably just ban them.
< gribble> https://github.com/bitcoin/bitcoin/issues/20234 | net: dont extra bind for Tor if binds are restricted by vasild · Pull Request #20234 · bitcoin/bitcoin · GitHub
< vasild> +1
< bitcoin-git> [bitcoin] laanwj pushed 4 commits to master: https://github.com/bitcoin/bitcoin/compare/924a4ff7eb62...f3727fd73538
< bitcoin-git> bitcoin/master 7b54d76 Luke Dashjr: Make sqlite support optional (compile-time)
< bitcoin-git> bitcoin/master 6608fec Luke Dashjr: GUI: Create Wallet: Nicely disable descriptor wallet checkbox if sqlite su...
< bitcoin-git> bitcoin/master bbb42a6 Luke Dashjr: RPC: createwallet: Nicer error message if descriptor wallet requested and ...
< bitcoin-git> [bitcoin] laanwj merged pull request #20156: build: Make sqlite support optional (compile-time) (master...opt_sqlite) https://github.com/bitcoin/bitcoin/pull/20156
< bitcoin-git> [bitcoin] laanwj pushed 5 commits to master: https://github.com/bitcoin/bitcoin/compare/f3727fd73538...2e2419711702
< bitcoin-git> bitcoin/master f8a1c4d Jon Atack: cli -netinfo: various quick updates and fixes
< bitcoin-git> bitcoin/master 33e9874 Jon Atack: cli -netinfo: make age column variable-width
< bitcoin-git> bitcoin/master 773f4c9 Jon Atack: cli -netinfo: handle longer tor v3 local addresses
< bitcoin-git> [bitcoin] laanwj merged pull request #20115: cli: -netinfo quick updates/fixups for 0.21 (master...netinfo-fixups) https://github.com/bitcoin/bitcoin/pull/20115
< queip> fanquake: FYI this is same guy that in past days is running around various IRC channels, threatens to kill Bitcoin developers. He rambles about Zeronet (which he ~falsely claims to be top developer of), about politics, bitcoin wiki, uses VPN, switches nicks, runs some scam foundation for "human rights"
< wumpus> oh no
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/2e2419711702...5b82f253b69f
< bitcoin-git> bitcoin/master d0a829e fanquake: build: fix mutex detection when building bdb on macOS
< bitcoin-git> bitcoin/master 5b82f25 Wladimir J. van der Laan: Merge #20195: build: fix mutex detection when building bdb on macOS
< bitcoin-git> [bitcoin] laanwj merged pull request #20195: build: fix mutex detection when building bdb on macOS (master...bdb_xcode12_implicit_function_decleration) https://github.com/bitcoin/bitcoin/pull/20195
< bitcoin-git> [bitcoin] fanquake pushed 3 commits to master: https://github.com/bitcoin/bitcoin/compare/5b82f253b69f...8e9e190ea5ce
< bitcoin-git> bitcoin/master 6c0259f Pieter Wuille: Squashed 'src/secp256k1/' changes from c6b6b8f1bb..3967d96bf1
< bitcoin-git> bitcoin/master 5803f5f Pieter Wuille: Update secp256k1 subtree to latest master
< bitcoin-git> bitcoin/master 8e9e190 fanquake: Merge #20257: Update secp256k1 subtree to latest master
< bitcoin-git> [bitcoin] fanquake merged pull request #20257: Update secp256k1 subtree to latest master (master...202010-secp256k1) https://github.com/bitcoin/bitcoin/pull/20257
< luke-jr> ryanofsky: I have no idea what you're missing in https://github.com/bitcoin/bitcoin/pull/20205#issuecomment-718758891
< luke-jr> ryanofsky: it's not an unrecognised row, it's MISSING ENTIRELY
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/8e9e190ea5ce...42b66a6b814b
< bitcoin-git> bitcoin/master 01476a8 Russell Yanofsky: wallet: Make -wallet setting not create wallets
< bitcoin-git> bitcoin/master 42b66a6 MarcoFalke: Merge #20186: wallet: Make -wallet setting not create wallets
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #20186: wallet: Make -wallet setting not create wallets (master...pr/nowa) https://github.com/bitcoin/bitcoin/pull/20186
< real_or_random> "we are announcing that travis-ci.org will be officially closed down completely no later than December 31st, 2020, allowing us to focus all our efforts on bringing new features and fixes to travis-ci.com "
< MarcoFalke> Finally. No travis builds means no travis failures !!11!
< luke-jr> sigh
< luke-jr> "Travis CI will continue to offer a free tier for public or open-source repositories on travis-ci.com and will not be affected by the migration."
< luke-jr> so it sounds like this is just consolidating platforms?
< real_or_random> I think consolidating is the goal
< real_or_random> but I don't understand the FAQ entry on write access
< real_or_random> will they have write access or not?
< luke-jr> sounds like "yes, but we promise not to use it, and are trying to avoid needing it by getting more granular permissions from GitHub"
< real_or_random> indeed
< real_or_random> this seems to affect the secp256k1 builds right now... apparently they're migrating the repos https://twitter.com/Mbussonn/status/1320748225838469121
< luke-jr> we probably trust GitHub's security stuff too much anyway4
< real_or_random> even though this page has instructions for migrating manually, and doesn't mention that they'll do it for us. And they can't because they need the permissions?
< luke-jr> is there a post-fetch hook to verify the merge signature?
< real_or_random> I don't understand what's going on. Maybe they're just moving their server so .com, so there are not many ressources for .org left, and that's why the builds ar slow
< real_or_random> when I try it for a local fork, it says
< real_or_random> " Read access to code, metadata, and pull requests "
< real_or_random> " Read and write access to checks, commit statuses, deployments, and repository hooks "
< real_or_random> I think I understand now... it's a mess
< real_or_random> The Travis "GitHub App" (which can be installed on repo) does not require read/write access
< wumpus> heh that sounds overly dramatic for basically just a rename
< real_or_random> The Travis Github OAuth thing (that you need for logging in on Travis) requires full read/write access to all repos you own.
< real_or_random> or not. it only says " Full control of private repositories"
< real_or_random> we may want to reopen https://github.com/bitcoin/bitcoin/issues/17802
< bitcoin-git> [gui] luke-jr opened pull request #125: GUI: Enable changing the autoprune block space size in intro dialog (master...intro_prune_size) https://github.com/bitcoin-core/gui/pull/125
< bitcoin-git> [bitcoin] achow101 opened pull request #20262: tests: Skip --descriptor tests if sqlite is not compiled (master...tests-check-sqlite) https://github.com/bitcoin/bitcoin/pull/20262
< luke-jr> #proposedmeetingtopic allowing sqlite wallet regression into 0.21
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #20263: Update assumed chain params (master...2010-21assumed) https://github.com/bitcoin/bitcoin/pull/20263
< bitcoin-git> [bitcoin] achow101 closed pull request #20260: wallet: Create named SQLite wallet files instead of wallet directories (master...single-file-sqlite) https://github.com/bitcoin/bitcoin/pull/20260
< hebasto> meeting?
< MarcoFalke> meeting?
< jonasschnelli> meeting!
< luke-jr> wumpus is late! :P
< MarcoFalke> I volunteer jonasschnelli
< jnewbery> Marco volunteers, jonasschnelli
< luke-jr> :D
< jonasschnelli> so shall it be...
< jonasschnelli> #startmeeting
< emzy> hi
< achow101> hi
< amiti> hi
< jonasschnelli> lightningbot?
< hebasto> hi
< ariard> hi
< jonasschnelli> #startmeeting
< hebasto> bad bot...
< achow101> maybe it's dead
< jonasschnelli> he refuses to work for me...
< MarcoFalke> #startmeeting
< jonasschnelli> however,... lets start anyways
< jonasschnelli> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb moneyball kvaciral ariard digi_james
< jonasschnelli> amiti fjahr jeremyrubin lightlike emilengler jonatack hebasto jb55 elichai2
< jonasschnelli> no proposed topics today I guess.
< jamesob> hi
< MarcoFalke> two topics
< jonasschnelli> any last-minute ones?
< achow101> luke-jr: proposed one
< MarcoFalke> [11:02] <jnewbery> #proposedmeetingtopic Disable S390x travis build
< MarcoFalke> [19:46] <luke-jr> #proposedmeetingtopic allowing sqlite wallet regression into 0.21
< jnewbery> jonasschnelli: I had one: Disable S390x travis build
< jonasschnelli> #action someone should update the channel topic: Meeting topics https://gist.github.com/moneyball/071d608fdae217c2a6d7c35955881d8a
< luke-jr> but it's not updated?
< jonasschnelli> Okay... lets start with 0.21 milestone first
< jonasschnelli> #topic 0.21 milestone
< jonasschnelli> luke-jr: yeah. someone should change that link
< luke-jr> my topic fits into this one :p
< luke-jr> #20205 had the milestone removed, but should be a blocker
< gribble> https://github.com/bitcoin/bitcoin/issues/20205 | wallet: Properly support a wallet id by achow101 · Pull Request #20205 · bitcoin/bitcoin · GitHub
< jonasschnelli> luke-jr: can you elaborate why it is a blocker? What is broken without it?
< luke-jr> jonasschnelli: without it, sqlite wallets are missing a unique id
< jonasschnelli> luke-jr: what user function does it break?
< luke-jr> jonasschnelli: BDB did this for us, but sqlite does not
< MarcoFalke> why would they need one?
< luke-jr> jonasschnelli: not having a unique id is a regression from the old wallets
< hebasto> is ryanofsky here?
< luke-jr> MarcoFalke: to distinguish between different wallets and renames/moves/copies
< achow101> the question is what unique ids do for us and whether not having one is a regression
< achow101> because we don't use them for anything currently
< luke-jr> achow101: Knots does
< jonasschnelli> have we expose the unique ID over our APIs or in the GUI?
< achow101> jonasschnelli: no
< luke-jr> it's an existing and long standing wallet feature
< luke-jr> also, we do up until now use them to disallow opening the same wallet twice
< achow101> it's a "feature" that's a byproduct of bdb and was only used by the bdb handling previously
< sipa> is it a feature, or a necessary restriction?
< jonasschnelli> I see luke-jr point... seems like a Knots issue too me. Core has never published or promised unique wallet id... however,.. seems to be easy to fix
< luke-jr> sipa: the unique id is certainly a feature
< achow101> I have no strong opinion either way, just that the way the luke-jr originally proposed was not the right way to add an id
< jonasschnelli> A uuid per wallet seems neat and useful. I agree with luke-jr
< luke-jr> jonasschnelli: I can't fix it in Knots alone; I would have to simply remove all functionality that uses it
< sipa> i tend to agree that it's a potentially useful feature
< sipa> but i don't know if it's a feature right now we need to make promises about
< jonasschnelli> Lets keep an eye on #20205...
< gribble> https://github.com/bitcoin/bitcoin/issues/20205 | wallet: Properly support a wallet id by achow101 · Pull Request #20205 · bitcoin/bitcoin · GitHub
< luke-jr> adding it doesn't require any promioses.
< sipa> sqlite wallets will work fine if you load 2 at the same time
< luke-jr> absolute worst case, we'd just not use it
< jonasschnelli> unsure if it is a 0.21 bugfix or a 0.22 feature.
< luke-jr> jonasschnelli: it's a regression
< sipa> luke-jr: if we never end up using it, it's hard to call it a bugfix now
< jnewbery> luke-jr: why do you say you can't fix it in knots?
< jonasschnelli> I guess adding it in 0.22 when we have 0.21 users creating sqlite wallets is not ideal
< luke-jr> sipa: the wallet is losing a feature
< emzy> I think if we have an ID we should use someting that is already pressent. Manybe the first public key.
< sipa> luke-jr: so what? it's a new type wallet; it has no features except the ones we say it has
< achow101> emzy: "first" is undefined, especially for descriptor wallets
< jonasschnelli> I guess having sqlite wallets without ID and some with may lead to bugs.
< luke-jr> jnewbery: 1) Core wallets will be missing it, and the same problems persist as adding ti later
< sipa> anyway, i'll comment on thePR
< luke-jr> jnewbery: 2) if I do anything to the wallet format in Knots, Core historically refuses to consider compatibility with it
< sipa> i'm in favor of having it, but i don't think it's fair to call it a bugfi
< sipa> luke-jr: obviously
< jonasschnelli> I added the 0.21 milestone to 20205
< jonasschnelli> lets continue on GitHub about whether its a bugfix or not
< sipa> no offense, but what you do in knots isn't relevant in this discussion
< luke-jr> sipa: see the question I am answering
< sipa> ok
< jonasschnelli> anything else that is relevant for the 0.21 milestone?
< luke-jr> for 0.20, we backported a fix that never got merged to master
< MarcoFalke> jonasschnelli: Only the things that are tagged
< sipa> luke-jr: oh?
< jonasschnelli> please help review 0.21: https://github.com/bitcoin/bitcoin/milestone/45 thanks
< luke-jr> part of #18818
< MarcoFalke> luke-jr: Yeah, at least we should cherry-pick what was backported
< gribble> https://github.com/bitcoin/bitcoin/issues/18818 | Fix release tarball generated by gitian by luke-jr · Pull Request #18818 · bitcoin/bitcoin · GitHub
< luke-jr> yes
< sipa> agree
< luke-jr> it would be ideal to just get it all merged in tho :p
< MarcoFalke> Though, it isn't a clean cherry-pick IIRC
< luke-jr> GitHub doesn't say there's conflicts O.o
< jonasschnelli> #topic Disable S390x travis build (jnewbery)
< jnewbery> The S390x builds on travis are very flakey and often fail, presumably because of contention issues at travis
< luke-jr> (should we also talk about the travis-ci migration?)
< sipa> is it just s390x?
< sipa> all of travis is pretty much broken for the secp256k1 repo
< luke-jr> :/
< sipa> s390x was the first sign, but soon after, everything stopped
< jonasschnelli> yeah.. I personally tend to ignore the CI icons ons pulls... its getting non-useful
< jnewbery> they were added to test under a big-endian platform. I'm not sure how many bugs have been caught by adding it, but I don't think it's worth keeping it given the costs of having to frequently rerun failed builds
< emzy> Is someone actually using bitcoin core on x390x?
< sipa> let me check if that's still the case
< luke-jr> emzy: it's the only Big Endian platform Travis had
< sipa> emzy: no, it's just to guarantee big-endian compatibility
< jnewbery> s390x seems to be the least reliable
< luke-jr> jnewbery: we could have a bot that restarts s390x if the rest pass?
< luke-jr> but should we wait and see if .com has this issue?
< sipa> (which, even if nobody actually uses bitcoin core on a big endian platform, is a good test for the correctness of the code, as it tends to expose implemention-dependency that isn't otherwise exposed)
< MarcoFalke> s390x found at least one bug, so I think it is useful
< luke-jr> Travis is only giving us 1 month to move, so..
< jonasschnelli> CIs are loosing their value if there are a large percentage of false "positives" (false fails)
< sipa> jonasschnelli: indeed, rapidly
< sipa> luke-jr: is power9 big endian?
< sipa> (typically)
< luke-jr> sipa: it's bi-endian, but Travis only supports LE afaik
< jnewbery> MarcoFalke: was that a new bug that was introduced, or one that had been around for a long time?
< sipa> luke-jr: on your power9 system, is the OS LE or BE?
< luke-jr> sipa: I run LE myself
< MarcoFalke> jnewbery: both. So it was two bugs actually
< jonasschnelli> I still recommend to continue build run our own CI system. bitcoinbuilds.org is a great start, ... runs more or less smooths since months,... can be easly extended and is cheeper than travis
< hebasto> and quicker
< MarcoFalke> jonasschnelli: We need integration with GitHub, otherwise no one will notice a failing build or even check them
< jonasschnelli> MarcoFalke: I have it
< jonasschnelli> Just not on the master branch....
< jonasschnelli> Will add it to the GUI repo soon
< jonasschnelli> It's not complicated
< luke-jr> who runs bitcoinbuilds.org?
< jonasschnelli> luke-jr: me
< MarcoFalke> Also, we need to be able to modify the config in-tree, not out-of-tree
< jonasschnelli> Yes. I'll add that soon.
< sipa> jonasschnelli: what architecture(s) does it support?
< jonasschnelli> Not saying it can replace travis,... but it may show the road to a successful and fast CI which we can easly maintain ourselfs
< luke-jr> can we make it use the GCC compile farm? :P
< jonasschnelli> sipa: its using libvirt under the hood
< MarcoFalke> Can it run qemu-s390x?
< luke-jr> jonasschnelli: that doesn't answer.. :P
< MarcoFalke> I think it is plain amd64 architecture
< jonasschnelli> Currently it can only run ubuntu18,... but we could add other servers with other architectures or qemu others (slow)
< luke-jr> GCC's farm has also ppc64, aarch64, sparc64, and mipsel
< luke-jr> jonasschnelli: can it work with just a shell?
< jonasschnelli> luke-jr: how do you mean that? no web frontend?
< luke-jr> jonasschnelli: I mean SSH into GCC's farm and run builds there
< jonasschnelli> would be possible...
< jonasschnelli> contribution welcome
< luke-jr> another project is already using it for their CI
< sipa> which project?
< luke-jr> "HansLambermont uses the compile farm for continuous build integration of the Stellarium project."
< jonasschnelli> as said... running and expanding our own CI seems to me the best way forward to finally get back to a CI state that is useful
< luke-jr> "Stellarium is a free open source planetarium for your computer. It shows a realistic sky in 3D, just like what you see with the naked eye, binoculars or a telescope."
< jonasschnelli> Our demand of funcionability goes beyond most CI product offerings,...
< jnewbery> I find it hard to believe that there isn't a reliable CI service that we can use and that we need to invent our own
< emzy> jonasschnelli: I'm open to help with it.
< jonasschnelli> emzy: great to hear.
< sipa> jnewbery: indeed...
< jonasschnelli> jnewbery: me too.. but our demand is non-normal
< sipa> does jenkins still exist?
< jonasschnelli> self hosted?
< MarcoFalke> jnewbery: All CI services are "lol we only support docker"
< sipa> MarcoFalke: is that a problem?
< MarcoFalke> sipa: Doesn't jenkins count as running our own?
< sipa> MarcoFalke: yes
< sipa> there is a different between running our own and inventing our own :)
< MarcoFalke> sipa: They don't run sanitizers or wine or ...
< jonasschnelli> sipa: it's already invented. It runs since 1 year
< jonasschnelli> (but missed additional features)
< jonasschnelli> *misses
< sipa> MarcoFalke: i'm confused what that has to do with docker
< sipa> sure you can find a docker image that contains sanitizer-enabled compilers and/or wine?
< MarcoFalke> sipa: You'll have to start the docker daemon with additional permissions
< luke-jr> so as far as Travis goes, are we just not migrating?
< sipa> i think we should keep our options open
< MarcoFalke> sipa: On travis we can start our own docker daemon
< MarcoFalke> luke-jr: I'd be surprised if travis improved by changing the domain name
< sipa> MarcoFalke: ok, i probably don't know enough about docker then... i'm not sure what that means or implies
< luke-jr> MarcoFalke: well, .com has historically been their commercial product?
< sipa> things could improve with travis.com... i'm not sure
< sipa> a question is if that requires giving them write permission to the repo, are we open to doing so?
< MarcoFalke> We used to send them money and support was still ignoring requests
< luke-jr> sipa: apparently it's write permissions to the user, not the repo specifically :/
< sipa> luke-jr: we can create a dummy bitcoin-core-ci user or something, no?
< luke-jr> MarcoFalke: they responded to me (maybe not timely or helpfully, I forget the details)
< luke-jr> sipa: maybe, but I think that violates GitHub's ToS
< MarcoFalke> luke-jr: well response is "lol, we don't support that" or "we'll discuss this internally"
< luke-jr> MarcoFalke: ☺
< MarcoFalke> luke-jr: Then DrahtBot would violate GitHub's ToS
< jonasschnelli> Migrations to .com makes sense to me as we have no alternative ready that has the same depths of testing... and it might get better (or worse?)
< jonasschnelli> back to the topic.... should we disable S390x?
< luke-jr> MarcoFalke: aha, they have an exception for bots, nm
< MarcoFalke> jonasschnelli: ACK
< jonasschnelli> ack
< MarcoFalke> jonasschnelli: Well, we could move it to cirrus ci for now
< sipa> if it doesn't work, it should be disabled
< hebasto> ack on disabling
< luke-jr> MarcoFalke: Cirrus has BE? :P
< MarcoFalke> in qemu
< luke-jr> >_<
< MarcoFalke> They are working on getting native arm though
< * luke-jr> should try getting x86_64 gitian working on his ppc64 host via qemu…
< luke-jr> (apparently I will have to patch qemu)
< sipa> even just doing BE builds in qemu for master (and not for every PR) would be a major win over not having it at all
< sipa> same with some other platforms
< MarcoFalke> I'll create a pull to move it to cirrus
< sipa> that'd be great
< jonasschnelli> Indeed. Building **everything** per each PR push is crazy
< jonasschnelli> (but great if it works)
< jnewbery> why is it crazy? It seems optimal if it works
< luke-jr> don't agree on crazy :P
< jonasschnelli> s/crazy/crazy-system-demanding
< sipa> if we have the resources, sure
< jonasschnelli> and it seems like we don't have the resources because large percentages of the builds fail for no reason
< jnewbery> cpu time is cheaper than developer time
< luke-jr> fail isn't a resource issue
< luke-jr> resource shortage would just mean long waits
< sipa> indeed
< jonasschnelli> jnewbery: agree... but if you can't be sure whether you have to debug the build or if it was just a glitch in the CI,... its a waste of time
< sipa> we're not low on resources, it's just broken, and we need to find something that works
< sipa> whatever we find is better than something that's broken
< luke-jr> ideally CI would prioritise builds, but otherwise include everything
< MarcoFalke> I don't think resources are the problem. The problem is the lack of ci infrastructure to trigger builds on working hardware, and someone who maintains the ci infrastructure.
< luke-jr> eg, put master above PRs
< luke-jr> maybe have a button where someone can say "I'm actually waiting on this"
< jonasschnelli> we need a CI-infrastructur-maintainer *duck*
< luke-jr> jonasschnelli: I thought you just volunteered? :P
< MarcoFalke> I volunteer jonasschnelli
< jonasschnelli> no way...
< jonasschnelli> it needs to be some linux/server crack
< luke-jr> [19:22:22] <luke-jr> who runs bitcoinbuilds.org? [19:22:26] <jonasschnelli> luke-jr: me
< luke-jr> we'll say that counts
< jnewbery> MarcoFalke: I agree. The problem is that we don't have a build manager
< jonasschnelli> I start stuff,... but not finish them
< luke-jr> XD
< sipa> jonasschnelli: i fear this will be a problem if we'd migrate to bitcoinbuilds.org for part or all of our CI
< sipa> things will break, and will require maintenance
< wumpus> oh crap don't tell me i missed the meeting almost because of the DST change here *facepalm*
< jonasschnelli> sipa: we should certenly not migrate before a group of people have commited time to it
< luke-jr> wumpus: lol
< sipa> wumpus: still 17 minutes left!
< jnewbery> I think all these problems are solvable by someone who has the time and desire to figure out the problems. Part of that might be researching different CI options
< MarcoFalke> wumpus: lol
< jonasschnelli> wumpus: hah
< sipa> wumpus: DST is evil
< wumpus> sipa: yess
< luke-jr> wumpus: nobody will ever know: the bot is broken to
< luke-jr> too*
< jonasschnelli> Ideally we have a team of people willing to maintain the CI
< MarcoFalke> jnewbery: I did look at all available ci options a year ago
< jonasschnelli> If it is all on my shoulders,.. it will break for sure
< luke-jr> jonasschnelli: like we don't have our hands full maintaining Core
< MarcoFalke> As I said most of them are docker-only, so they can't run our scripts
< dongcarl> If we have actual needs beyond what CI providers offer, then we should pay someone to maintain a CI cluster, no?
< jonasschnelli> luke-jr: there are enought people with server admin skills but no coding skills willing to contribute.
< luke-jr> dongcarl: that implies we have excess funding
< MarcoFalke> Only cirrus ci and travis offer full vms
< jonasschnelli> dongcarl: sure. Paing for it should be possible
< sipa> funding can be found, i'm sure
< jnewbery> I expect funding will not be a problem
< luke-jr> oh, so it's only my funding that's a problem :/
< luke-jr> sigh
< yanmaani> luke-jr: Are build servers *that* expensive?
< jonasschnelli> no... but the maintenance men.hours
< luke-jr> on that note, funding might be good for security bug bounties too
< luke-jr> yanmaani: no, but maintaining a CI platform probably is
< yanmaani> oh right, misread
< emzy> I think a CI team is better than a single persion.
< jnewbery> *or woman hours :)
< jonasschnelli> emzy: for sure
< jonasschnelli> jnewbery: oops. yes.
< MarcoFalke> Have people had issues with Cirrus Ci lately?
< luke-jr> MarcoFalke: I started ignoring it because when it does have problems I can't restart it
< sipa> MarcoFalke: i can't remember any... i also have no idea what actually runs on it
< MarcoFalke> They used to abort sometimes, but setting kvm:true fixed it I think
< MarcoFalke> sipa: Only the thread sanitizer and one fuzzer
< sipa> is there any reason why everything couldn't run on cirrus?
< MarcoFalke> We could also move all jobs to Cirrus Ci, which would probably mean we need to pay them
< MarcoFalke> free tier is only 8 cpus or so
< luke-jr> probably cheaper than maintaining our own
< luke-jr> if it works decently
< MarcoFalke> probably
< dongcarl> Do we have any needs beyond what Cirrus provides?
< jonasschnelli> Semaphore offered us also free service https://semaphoreci.com
< MarcoFalke> dongcarl: s390x and arm would be nice, but we can use qemu
< luke-jr> jonasschnelli: but that's just Docker?
< MarcoFalke> jonasschnelli: If it is just docker, it won't work
< jonasschnelli> luke-jr: they have x86 and macOS... all docker I guess
< MarcoFalke> If someone knows another non-docker ci, let me know, but I don't think there is
< luke-jr> MarcoFalke: hmm, does KVM work inside stock Docker?
< bitcoin-git> [bitcoin] gwillen opened pull request #20264: test: Make secp tests optional in `make check` (master...feature-optional-secp-check) https://github.com/bitcoin/bitcoin/pull/20264
< MarcoFalke> luke-jr: we don't use kvm
< MarcoFalke> I doubt it does
< MarcoFalke> qemu-user works in docker
< luke-jr> MarcoFalke: we could
< MarcoFalke> luke-jr: Patches welcome :)
< MarcoFalke> I don't have background in kvm, so I can't help here
< luke-jr> MarcoFalke: without a plan, no point
< luke-jr> MarcoFalke: KVM is just virtualising a machine - so you can do anything as if it was real hardware
< MarcoFalke> luke-jr: git grep docker ./ci
< MarcoFalke> then replace docker with kvm
< luke-jr> sure, but unless Docker supports KVM, it won't help
< MarcoFalke> jup, I don't know an answer to that
< jonasschnelli> Other topics?
< jonasschnelli> (short ones)
< jonasschnelli> #endmeeting
< jonasschnelli> \o
< jnewbery> thanks jonasschnelli!
< luke-jr> wumpus: okay, you can start your meeting now :P
< jonasschnelli> hehe
< wumpus> yes, thanks for picking it up jonasschnelli
< dongcarl> MarcoFalke: can we do s390x qemu on Travis x86? Or way too slow for that?
< MarcoFalke> dongcarl: we can. It should only be a bit slower
< wumpus> it would be more efficient to do the build using cross-compile, then run only the unit tests using qemu
< MarcoFalke> wumpus: That is what it does
< MarcoFalke> So compilation is the same cost
< wumpus> ok, didn't realize that,I thought it did the whole build on s390s
< sipa> i thought it was actually running on s390x hardware
< wumpus> qemu-user is *weird* with differnt endianness though
< MarcoFalke> sipa: Right now it does (well if it worked)
< sipa> MarcoFalke: oh ok
< sipa> so what is the "That is what it does" about?
< emzy> Is qemu acurate enough to really find problems with BE?
< MarcoFalke> Though, we can toggle one line in the config to use qemu-user
< sipa> emzy: definitely
< wumpus> qemu system emulation is
< * dongcarl> wishes Travis had binfmt_misc qemu images
< wumpus> qemu user emulation is *not* for swapped endianness
< MarcoFalke> sipa: If the ci system were using qemu, it would cross compile and only run qemu-user
< MarcoFalke> wumpus: Oh, it is not? I didn't know that
< wumpus> it fails to swap some linux kernel structures
< wumpus> maybe it's better today but there used to be quite some problems making the platform much stranger than an actual big-endian machine :-)
< MarcoFalke> I guess the question is: will qemu-user be able to find endian issues in the Bitcoin Core codebase
< MarcoFalke> We don't care about issues in the linux kernel
< wumpus> it found some false positives at the time
< luke-jr> qemu-system has trouble with x86 on ppc due to the memory model differences :x
< wumpus> no, but it communicates with the linux kernel
< sipa> this should be easy to figure out
< sipa> if qemu-user works with s390x
< wumpus> every field in every structure that's directly or indirectly exchanged needs to be endian-corrected
< wumpus> that's not a trivial task and error prone
< luke-jr> whatever CI we go with, couldn't we just buy them a POWER system and run it in BE? :P
< luke-jr> rather than trying to sort out emulation mess
< wumpus> which is why I doubt whether it's a good reflection of actual hardware a bit
< MarcoFalke> luke-jr: They use google cloud and packet or amazon. Good luck buying them a POWER system
< luke-jr> (actually, POWER8/9 should be capable of running VMs in both endians..)
< luke-jr> MarcoFalke: ugh
< jonatack> wumpus: arf i missed it too due to DST * makes note
< wumpus> jonatack: hah better next week
< jonatack> hehe
< emzy> wumpus online calendar worked perfectly for me. :)
< wumpus> emzy: glad to hear that :-)
< luke-jr> KOrganizer 4.4 here <.<
< emzy> Apple Calendar here <.<
< wumpus> in any case I don't have anything against a qemu-user based solution for BE if it actually works now and doesn't require weird workarounds in the tests
< MarcoFalke> wumpus: I'll test :)
< wumpus> MarcoFalke: thanks!
< sipa> let's see how far i get with qemu-user s390x in a debian vm
< bitcoin-git> [bitcoin] mjdietzx opened pull request #20265: refactor: get wallet path relative to wallet_dir (master...refactor-get-relative-wallet-path) https://github.com/bitcoin/bitcoin/pull/20265
< sipa> segfault when running a hello world :(
< sipa> it works with aarch64
< bitcoin-git> [bitcoin] mjdietzx closed pull request #20265: refactor: get wallet path relative to wallet_dir (master...refactor-get-relative-wallet-path) https://github.com/bitcoin/bitcoin/pull/20265
< wumpus> good to know, yes i'd expect aarch64 on amd64 to be very well tested
< jonasschnelli> I'm even failing setting up debian stretch s390x with qemu-system-s390x...
< sipa> going to try with testing
< sipa> wumpus: hello world works in debian testing!
< sipa> $ s390x-linux-gnu-gcc test.c -o test
< sipa> # qemu-s390x-static test
< sipa> hello world
< real_or_random> I don't see the full log here for some reason but people saw https://github.com/bitcoin-core/secp256k1/pull/843 ?
< sipa> (the $/# is a typo, i don't have copy paste from my VM enabled)
< real_or_random> everyone seems to use docker images for qemu on travis... that's easy and seems to be good enough for travis
< sipa> real_or_random: oh interesting
< real_or_random> I got this running locally within a few minutes
< sipa> i assumed it was using system qemu
< real_or_random> ocker run --rm --privileged multiarch/qemu-user-static --reset -p yes
< real_or_random> docker run --rm -t -v '/your/local/secpdir:/mnt/secp256k1' -w /mnt/secp256k1 aarch64/busybox:glibc ./tests
< real_or_random> these two commands do the job for me
< real_or_random> (after cross compiling)
< real_or_random> which is just ./configure --host=aarch64-ibm-linux-gnu
< sipa> does that also work locally?
< real_or_random> this works locally yes, should also work on travis but it failed because it did a stupid mistake
< real_or_random> (sorry, I pasted the aarch64 example, which is LE)
< real_or_random> this should work with s390x too, host=s390x-unknown-linux-gnu
< bitcoin-git> [bitcoin] achow101 opened pull request #20266: Fix change detection of imported internal descriptors (master...fix-desc-change) https://github.com/bitcoin/bitcoin/pull/20266
< real_or_random> I got a travis job on my own repo https://travis-ci.org/github/real-or-random/secp256k1/jobs/739959302#L620 ... it failed because I forgot to replace my local path in the command -.-
< wumpus> sipa: awesome!
< real_or_random> being able to run things on qemu is certainly useful but I think the bigger issue is the travis migration
< sipa> real_or_random: trying to get that to work locally
< sipa> running into weird zfs issues
< real_or_random> what kind of issues?
< sipa> apparently the ubuntu docker install creates zfs filesystems for each docker image or something, unsure
< sipa> and now there is one in use which it can't destroy
< sipa> which i don't understand as it's not mounted
< real_or_random> sipa: which command are you trying to run?
< sipa> real_or_random: the first ons, run --rm -t -v ...
< real_or_random> hm, yeah, no idea. that one run on successfully also on travis.
< sipa> real_or_random: ok, works
< sipa> i had to remove the "--rm"
< bitcoin-git> [bitcoin] achow101 opened pull request #20267: Disable and fix tests for when BDB is not compiled (master...tests-opt-sqlite-bdb) https://github.com/bitcoin/bitcoin/pull/20267