< bitcoin-git> [bitcoin] jkczyz opened pull request #17399: validation: Templatize ValidationState instead of subclassing (master...2019-11-validation-state) https://github.com/bitcoin/bitcoin/pull/17399
< sipa> bech32_tests.obj : error LNK2001: unresolved external symbol "bool __cdecl CaseInsensitiveEqual(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &)"
< sipa> is this a known issue?
< lightlike> sipa: yes, see #17384 - all appveyor builds are currently failing (except the one in 17384 :-) )
< gribble> https://github.com/bitcoin/bitcoin/issues/17384 | test: Create new test library by MarcoFalke . Pull Request #17384 . bitcoin/bitcoin . GitHub
< promag> Why "an"?
< real_or_random> promag: "nrequired" starts with a vowel sound
< real_or_random> try to pronounce it
< promag> real_or_random: oh thats a rule? ok then ?
< promag> ty
< jonatack> promag: yes, what real_or_random said. See discussion at https://github.com/bitcoin/bitcoin/pull/16047#discussion_r285628201
< jonatack> er, that link is the inverse case :)
< wumpus> I tend to get that wrong too, it's a very difficult rule for non native speakers of English
< jonatack> I agree it can look strange to read
< wumpus> for 30 years of so I was blissfully unaware and based a/an on the simple letter-based heuristic, it worked enough for people to never complain
< elichai2> Still have some problems but look at build no. 3 :) https://travis-ci.org/elichai/bitcoin/builds/608703220
< wumpus> elichai2: you've added a native ppc build? nice
< elichai2> yep. it's kinda undocumented, so we might need to allow it to fail (altough it works on my repos), still have some problems with the docker. but it's pretty cool
< wumpus> so they have actual POWER servers running builds? sounds expensive :)
< elichai2> wumpus: either that or a bunch of emulation stuff
< wumpus> for ARM it's not hard to believe it's real native
< luke-jr> POWER isn't that expensive either, especially for business..
< luke-jr> I built my Talos II for $3k and there's the cheaper Blackbird now
< luke-jr> wumpus: Technically you could always use "an" and just defer to the Tonal number system (where "an" is 1)
< elichai2> if any docker wizard has any inputs on what's going on here it'll be appretiated :) (added `set -x` and some docker ls's here and there) https://travis-ci.org/elichai/bitcoin/jobs/608713371#L435
< wumpus> sure, but it's more expensive than x86; ARM is really really cheap so adding that was kind of low-hanging fruit
< wumpus> heh yes
< wumpus> I don't know anything about docker, sorry
< luke-jr> wumpus: I'm sure Travis has way many more x86 than POWER though ;)
< luke-jr> (and POWER probably finishes the jobs faster to make up for it!)
< bitcoin-git> [bitcoin] MarcoFalke pushed 3 commits to master: https://github.com/bitcoin/bitcoin/compare/e65b4160e99f...46fc4d1a24c8
< bitcoin-git> bitcoin/master faec282 MarcoFalke: scripted-diff: test: Move setup_common to test library
< bitcoin-git> bitcoin/master fa4c6fa MarcoFalke: doc: Add documentation for new test/lib
< bitcoin-git> bitcoin/master 46fc4d1 MarcoFalke: Merge #17384: test: Create new test library
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #17384: test: Create new test library (master...1911-testLib) https://github.com/bitcoin/bitcoin/pull/17384
< wumpus> I don't get it, ci/lint/06_script.sh runs subtree checks for various trees, but doesn't import their master branches, how is this supposed to work?
< wumpus> I added a new subtree in #17398 and locally the subtree checks pass, but not on travis, because it doesn't have the commit available
< gribble> https://github.com/bitcoin/bitcoin/issues/17398 | WIP: build: Update leveldb to 1.22+ by laanwj . Pull Request #17398 . bitcoin/bitcoin . GitHub
< wumpus> ok added an explicit "git fetch https://github.com/google/crc32c.git master" for now, let's see...
< wumpus> #proposedmeetingtopic Move subtrees out of src/
< wumpus> or at least under a single roo
< wumpus> would be a lot easier for linters and doxugen and maintenance scripts and such if there was a single path to ignore
< wumpus> e.g. https://github.com/bitcoin/bitcoin/pull/17398/commits/b9b829574c517fcbc0fbcbda67ed667b04a73899 the other option would be to have an unified exclude path, but having to specify it 10+ times is absurd
< wumpus> YESS they pass
< fanquake> guess we can wait for a bunch of rebases now that #17384 went in
< gribble> https://github.com/bitcoin/bitcoin/issues/17384 | test: Create new test library by MarcoFalke . Pull Request #17384 . bitcoin/bitcoin . GitHub
< fanquake> Had been ready to merge 17342
< wumpus> same
< wumpus> ooh the native osx travis build has beer keg emojis, yes it's the first time I look at that
< fanquake> is that in the brew output?
< fanquake> We can turn that off if required heh
< wumpus> no, let's leave it
< fanquake> We can also customize it using HOMEBREW_INSTALL_BADGE
< wumpus> "checking for F_FULLFSYNC... no" looks like the detection is somehow not working, fullfsync is a MacOS thing right?
< wumpus> oh, I think I understand the issue
< fanquake> wumpus let me know when that PR is "stable" enough to test in various places
< wumpus> fanquake: sure; it should work now on linux/windows/osx, though without crc32c acceleration
< wumpus> integrating crc32c into the build system is going to be slightly involved, there's various new things (besides sse42 support) that need to be detected
< fanquake> wumpus ok. Will do some initial testing on macOS / BSDs
< wumpus> most of the fixes I've pushed in the last hour or so, except for the FULLFSYNC one, were to make linters happy
< wumpus> that's how it goes right ...
< fanquake> ?
< fanquake> I've been trying to track down the file descriptor discrepancy for the past day or so. Think it's time to give up and dump the info in an issue..
< wumpus> that sometimes helps, maybe someone else has ideas
< wumpus> cleaned up the commits; everything besides "build: Update build system for new leveldb" (and the subtree update for src/leveldb) is related to crc32c and not necessary for build at the moment
< fanquake> cool
< wumpus> (but also shouldn't get in the way; HAVE_CRC32C is hardwired to 0 at the moment)
< fanquake> ok
< fanquake> are we discussing 0.19.0 release at the meeting this arvo?
< wumpus> yes, we should
< wumpus> I think it's ready, nothing else has come up with the last rc
< fanquake> Yep.
< fanquake> One final thing we probably want to do is at least add a note to the release notes in regards to macOS Catalina users having to "right click" and open.
< fanquake> Might avoid some potential confusion.
< wumpus> yes, definitely
< fanquake> That's all I'll say in regards to macOS atm.
< fanquake> Can anyone remember in which PR sdaftuar mined a commit hash to have 6-7 leading 0s ?
< fanquake> or it might actually have been marco
< aj> fanquake: "git log 000000" should tell you, #13510
< gribble> https://github.com/bitcoin/bitcoin/issues/13510 | Scripts and tools: Obsolete #!/bin/bash shebang by DesWurstes . Pull Request #13510 . bitcoin/bitcoin . GitHub
< fanquake> aj cheers. Guess I completely mis-remembered
< aj> fanquake: marcofalke has a shorter one in #14264 by the looks
< gribble> https://github.com/bitcoin/bitcoin/issues/14264 | doc: Split depends installation instructions per arch by MarcoFalke . Pull Request #14264 . bitcoin/bitcoin . GitHub
< fanquake> aj thanks for both. 14264 doesn't have any ACK comments, so I'll use 13510
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/46fc4d1a24c8...46e0e276398e
< bitcoin-git> bitcoin/master 3c84dee Aaron Clauson: Updated appveyor config:
< bitcoin-git> bitcoin/master 46e0e27 MarcoFalke: Merge #17364: Updates to appveyor config for VS2019 and Qt5.9.8 + msvc pro...
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #17364: Updates to appveyor config for VS2019 and Qt5.9.8 + msvc project fixes (master...vs2019_oct31) https://github.com/bitcoin/bitcoin/pull/17364
< elichai2> FIxed the docker problem. I'm now getting `conftest.cpp:32:10: fatal error: sys/endian.h: No such file or directory`
< wumpus> sorry, your system is middle-endian so lacks the endian header :-)
< bitcoin-git> [bitcoin] MarcoFalke pushed 3 commits to master: https://github.com/bitcoin/bitcoin/compare/46e0e276398e...7d14e35f3fb7
< bitcoin-git> bitcoin/master d995762 John Newbery: [tests] Don't use TestingSetup in the checkqueue_tests
< elichai2> so that's fine? I guess that's not the actual error. there's too much noise, i'll remove the `set -x` now
< bitcoin-git> bitcoin/master 5506ecf John Newbery: [refactor] Replace global int nScriptCheckThreads with bool
< bitcoin-git> bitcoin/master 7d14e35 MarcoFalke: Merge #17342: refactor: Clean up nScriptCheckThreads
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #17342: refactor: Clean up nScriptCheckThreads (master...2019-11-nScriptCheckThreads-comment) https://github.com/bitcoin/bitcoin/pull/17342
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/7d14e35f3fb7...772673dfbe70
< bitcoin-git> bitcoin/master fae43a9 MarcoFalke: test: Seed test RNG context for each test case, print seed
< bitcoin-git> bitcoin/master 772673d MarcoFalke: Merge #16978: test: Seed test RNG context for each test case, print seed
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #16978: test: Seed test RNG context for each test case, print seed (master...1909-testSeed) https://github.com/bitcoin/bitcoin/pull/16978
< fanquake> #proposedmeetingtopic not so much a topic, but a reminder that if everyone wants to air / dump their GitHub grievances into #15847, I'll be discussing with GH next week
< gribble> https://github.com/bitcoin/bitcoin/issues/15847 | Feedback for GitHub CEO . Issue #15847 . bitcoin/bitcoin . GitHub
< elichai2> Getting frustrated. at this point any help would be wellcome. I can't figure out why this test is failing :/ https://travis-ci.org/elichai/bitcoin/jobs/608789848
< fanquake> elichai2 a test, or the depends build of OpenSSL ?
< fanquake> We use 'Configure' to configure OpenSSL in depends, looks like it's suggesting to use ./config instead. I'd assume Configure is bombing out for some reason
< elichai2> so why will it fail only on here?
< elichai2> (maybe the tests shouldn't use the depends and add the incompatible db flag?)
< fanquake> I'd guess no one has tested a depends build with a powerpc64le-unknown-linux-gnu HOST before, and haven't run into the OpenSSL configure failure
< elichai2> fanquake: what do you think, should I try fighting it (replacing Configure with `./config` or just not use the depends?)
< fanquake> elichai2 I'm going to spin up a container and take a quick look in a second.
< fanquake> In any case hopefully OpenSSL will be gone soon
< elichai2> k. i'll for now try without depends to see if any other erros come up that are more directly related to bitcoin
< elichai2> if you have a local container with ppc64le feel free to debug a Docker bug i've found that made me replace `library/ubuntu` with `ppc64le/ubuntu:18.04`
< fanquake> elichai2 I've recreated your issue locally with HOST=powerpc64le-unknown-linux-gnu. OpenSSLs ./Configure fails. ./config detects ppc64le-whatever-linux2 and will configure for linux-generic32
< elichai2> fanquake: does that mean it's working, `linux-generic32` doesn't sound good hehe
< elichai2> (it's still building with no depends https://travis-ci.org/elichai/bitcoin/jobs/608821982)
< fanquake> I assume linux-ppc64 is what we should be passing to ./Configure, but I don't know what detection is failing
< fanquake> that openSSL doens't figure that out.
< bitcoin-git> [bitcoin] fanquake pushed 3 commits to master: https://github.com/bitcoin/bitcoin/compare/772673dfbe70...270616228bc9
< bitcoin-git> bitcoin/master 6d50b26 Jon Atack: test: add logging to wallet_avoidreuse.py
< bitcoin-git> bitcoin/master 0e7c90e Jon Atack: test: speed up wallet_avoidreuse.py
< bitcoin-git> bitcoin/master 2706162 fanquake: Merge #17362: test: speed up wallet_avoidreuse, add logging
< elichai2> hmmm `/home/travis/build/elichai/bitcoin/build/bitcoin-powerpc64le-unknown-linux-gnu/src/compat/glibc_compat.cpp:75: undefined reference to `log2f_old'`
< bitcoin-git> [bitcoin] fanquake merged pull request #17362: test: speed up wallet_avoidreuse, add logging (master...wallet_avoidreuse-test-improvements) https://github.com/bitcoin/bitcoin/pull/17362
< elichai2> oh. we have no ifdef for powerpc
< elichai2> I thought luke-jr / BlueMatt are testing/using PowerPC
< elichai2> i'll try dropping `--enable-glibc-back-compat`
< BlueMatt> I work daily on ppc64, though I dont daily build bitcoin core
< BlueMatt> at least not latest master
< MarcoFalke> #14066 might fix the depends issue
< gribble> https://github.com/bitcoin/bitcoin/issues/14066 | gitian-linux: Build binaries for 64-bit POWER by luke-jr . Pull Request #14066 . bitcoin/bitcoin . GitHub
< MarcoFalke> elichai2: The functional tests are a bit slow. You might just disable them for now
< elichai2> MarcoFalke: you're saying the failure is a timeout?
< MarcoFalke> you 120 seconds rpc timeout
< MarcoFalke> *yeah,
< elichai2> it sucks that travis doesn't cache if the build fails :/ I need it to succeed at least once so i'll have faster builds lol
< elichai2> MarcoFalke: so I should not run them?
< MarcoFalke> You may also bump the timeouts
< elichai2> I guess that's a change to the test framework though?
< MarcoFalke> Idk. I kept bumping them for years
< elichai2> or is there a env variable?
< MarcoFalke> Maybe we should remove the timeout
< MarcoFalke> They are hardcoded in the individual test files
< elichai2> k. i'll start by making this work *without* functional. so I can open a PR and document what works and what doesn't, and then look into timeouts
< MarcoFalke> self.rpc_timeout = 120
< wumpus> elichai2: you're not basing it on luke-jr's PR? I thought that was the idea
< elichai2> :O
< elichai2> #14066?
< gribble> https://github.com/bitcoin/bitcoin/issues/14066 | gitian-linux: Build binaries for 64-bit POWER by luke-jr . Pull Request #14066 . bitcoin/bitcoin . GitHub
< wumpus> yes
< wumpus> would be nice to have that in for 0.20
< elichai2> It's been a year, hmm the only thing there that's related is the glibc_compact. But I assume the CI will increase people's confidence in that PR?
< wumpus> helping test and review it would increase confidence in the PR, just asking because you care about ppc apparently :)
< wumpus> and yes I don't think it makes sense to add a travis run for a platform we don't distribute binaries for
< wumpus> unless it's a big endian platform, would be nice to have a big-endian travis run no matter what
< sipa> wumpus: (not sure where to comment) your leveldb commit 180296c359ba248ae6f2a6094098a22fd31994d6 drops a few 'override' modifiers for no reason, i think
< wumpus> sipa: oh that's not intentional, thanks
< sipa> also in 415ad71a96070dd4989153a800e6fd969269590b's commit message you have a typo in your own name
< sipa> (i think!)
< elichai2> wumpus: I don't even have ppc, was just excited that I found this new Travis feature, and always looking for ways to contribute :)
< wumpus> sipa: thanks, will fix
< sipa> the patchset for leveldb seems amazingly small now
< sipa> and entirely upstreamable?
< wumpus> sipa: yes, I think it's eventually upstreamable
< sipa> this is great :)
< wumpus> already had to make a patch on top on crc32c too, build system related, https://github.com/bitcoin-core/crc32c/commit/a3f2c1c13051d922cef031aec505801462080579 but should be upstreamable too I think
< sipa> we're using the cmake buildsystem for crc32c?
< wumpus> no :)
< sipa> then why does your patch modify CMakeLists.txt?
< fanquake> Glad we don't have too
< wumpus> I've integrated crc32c into the build system in the same way as leveldb, this requires being able to pass defined manually instead of having this .h file
< wumpus> because cmake uses that .h.in file to provide configuration
< sipa> oh i see, to make sure things don't break for when people build the patched tree using cmake
< wumpus> I don't, but don't want to break their buid system...
< wumpus> yes
< sipa> of course
< wumpus> I have sse42 already working locally (it's not in the PR yet), now trying to get ARM64 crc32c to work
< wumpus> it's easy just have to convert the cmake detection to autoconf (it doesn't help I don't really know either very well though :)
< sipa> nobody does.
< wumpus> true.
< MarcoFalke> is the meeting in 9 minutes?
< fanquake> *8 minutes
< wumpus> I think so
< wumpus> date -u shows it's 18:53 so yea
< wumpus> #startmeeting
< lightningbot> Meeting started Thu Nov 7 19:00:29 2019 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
< kanzure> hi
< gleb> hi
< jonatack> bonsoir
< jnewbery> hi
< dongcarl> saluton
< digi_james> hi
< fanquake> hi
< provoostenator> hi
< wumpus> #topic 0.19.0 final
< meshcollider> hi
< sipa> hi
< wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb moneyball kvaciral ariard digi_james amiti fjahr
< wumpus> jeremyrubin lightlike
< jamesob> hi
< fanquake> #17079 #16996
< gribble> https://github.com/bitcoin/bitcoin/issues/17079 | v0.19.0 testing . Issue #17079 . bitcoin/bitcoin . GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/16996 | Release process for 0.19.0 . Issue #16996 . bitcoin/bitcoin . GitHub
< fanquake> Does anyone have new issues to report after testing rc3?
< wumpus> I haven't heard of any new problems coming up with rc3, so it's probably time to tag final?
< instagibbs> hi
< fanquake> ACK. I haven't seen anything new either.
< achow101> hi
< wumpus> well, first moving the release notes back to the branch from the wiki
< fanquake> and adding the macOS catalina note
< wumpus> so if anyone wants to do any last minute edits in the release notes, now would be the time
< MarcoFalke> ship it
< wumpus> yess
< fanquake> ?
< provoostenator> Ship it!
< arik_> ?
< wumpus> #topic High priority for review
< fanquake> I guess we could shift topics then?
< wumpus> 7 blockers, 7 things chasing ACK
< wumpus> I think that's already a perfect number so let's move on
< BlueMatt> proposed topic: one last question blocking rust progress
< wumpus> (anything to add/remove?)
< achow101> add #17373 pls
< gribble> https://github.com/bitcoin/bitcoin/issues/17373 | wallet: Various fixes and cleanup to keypool handling in LegacyScriptPubKeyMan and CWallet by achow101 . Pull Request #17373 . bitcoin/bitcoin . GitHub
< wumpus> achow101: ok, added
< fanquake> Only PR I'd suggest adding is #17270. I have some new comments to make there, and think it could be split up to aid review, but could use more eyes regardless.
< gribble> https://github.com/bitcoin/bitcoin/issues/17270 | Feed environment data into RNG initializers by sipa . Pull Request #17270 . bitcoin/bitcoin . GitHub
< gleb> I think there is some valuable discussion is going on here: #17326. Not sure it's a particularly high-prio, but I think it's worth looking at for whoever missed. It's about whether we should keep (some) p2p connections on a node restart or connect to new peers.
< gribble> https://github.com/bitcoin/bitcoin/issues/17326 | rfc, p2p: Eclipse attack mitigation . Issue #17326 . bitcoin/bitcoin . GitHub
< gleb> It's not even mine :P
< wumpus> it doesn't have to be to propose it
< wumpus> but yea, that would be more like a proposed meeting topic than something for review
< fanquake> Should we move onto a proposed topic? This might be a quick meeting for once.
< fanquake> is nickler or instagibbs here?
< gleb> I don't think at this point we need real-time discussion, as long as the conversation is happening there on github. Just wanted to bring a bit attention of everyone.
< wumpus> fanquake: I think everyone is already looking at #17270
< gribble> https://github.com/bitcoin/bitcoin/issues/17270 | Feed environment data into RNG initializers by sipa . Pull Request #17270 . bitcoin/bitcoin . GitHub
< wumpus> happy with adding that to high prio, though
< MarcoFalke> Also, wumpus had another topic
< wumpus> yes
< fanquake> wumpus: if everyone is looking at it, I'm surprised there hasn't been more commentary
< wumpus> quite a lot of topics this week, let's move on
< wumpus> #topic security email address for bitcoin-core/secp256k1 (nickler)
< sipa> real_or_random, nickler: here?
< nickler> hi
< BlueMatt> seems like an easy solution is proposed there: add a list for secp-securiy@bitcoincore.org and add relevant folks to it?
< nickler> We're adding a SECURITY.md file to the bitcoin-core/secp256k1 repo and considered secp256k1-security@bitcoincore.org as security email address.Would it be possible to add such an email address? How are we making sure that security contacts will actually get emails forwarded to them? Who can see the content of a vulnerability report if it's sent unencrypted?
< BlueMatt> I presume wumpus can do that?
< wumpus> yes, I can make a forward address on bitcoin-core.org
< BlueMatt> dig +short bitcoincore.org mx
< BlueMatt> 10 spool.mail.gandi.net.
< wumpus> just tell me who to include
< wumpus> doesn't have to be in the meeting :p
< nickler> wumpus: cool, will do in private
< sipa> we'd need to put it on the bitcoincore.org website as well, i think
< sipa> together with gpg keys of some/all of the people involved
< wumpus> sure
< BlueMatt> sounds good, next topic?
< bitcoin-git> [bitcoin] elichai opened pull request #17402: Travis support for PowerPC64 (master...2019-11-powerpc64) https://github.com/bitcoin/bitcoin/pull/17402
< wumpus> #topic opt-in SRD (instagibbs)
< wumpus> #17331
< gribble> https://github.com/bitcoin/bitcoin/issues/17331 | Use effective values throughout coin selection by achow101 . Pull Request #17331 . bitcoin/bitcoin . GitHub
< achow101> instagibbs wanted to ask whether we should consider making SRD something that is opt in instead of outright replacing the current knapsacksolver fallback
< wumpus> could at least initially do that I guess
< gleb> Is it assumed that everybody knows what does SRD stand for?
< achow101> so people who care could set some switch to use srd as the fallback so that we aren't breaking things too much
< sipa> gleb: single random draw
< wumpus> replacing it is goig to be much more controversial than adding a new method
< achow101> but I don't think anyone would opt in because what user knows that srd is or really cares about the coin selection algo details?
< achow101> My current pr for srd is to actually use both knapsack and srd, then choose the one that produces the "better" solution
< wumpus> it's something that needs to be tested/evaluate over a long period
< achow101> although defining "better" is non-trivial
< wumpus> people that want to test it can enable it
< achow101> i'm not convinced that people are going to test it
< wumpus> you mean, no one?
< meshcollider> Probably lol
< wumpus> that's a dangerous statement, if you think no one cares, why work on it?
< sipa> a possibility could be to run both, and add a debug log that tells you what SRD would have done instead
< achow101> I would guess that the people who would test it are not the same people who make enough transactions for it to matter
< wumpus> that doesn't matter
< sipa> there's a big difference between people in general being interested in a feature ("better coin selection" i think certainly qualified) and people with both the technical means and in a situation where they can provide good feedback
< jonatack> perhaps it needs awareness raised about it? via bitcoin optech, forums, etc.
< achow101> then there's no meaningful results or feedback
< wumpus> though it might mean you need to convince people that make enough transactions to test it, maybe
< wumpus> that's easier if it's an option
< wumpus> I don't think we can have this discussion without anyone arguing to add it
< wumpus> let's move to next topic
< wumpus> #topic Subtree organization and linters (wumpus)
< fanquake> linters ?
< wumpus> so while trying to add crc32c to the tree (because the new leveldb uses that), I've noticed that the exceptions to linting are spread all over the place
< wumpus> I think it'd be useful to move subtrees to one place in the tree
< sipa> yes!
< MarcoFalke> Linters shouldn't be a reason to move source code around
< BlueMatt> ack
< wumpus> so that doxygen knows to avoid it, linters know to avoid it, etc
< gleb> broke: minting; woke: linting
< MarcoFalke> But I agree it makes sense even absent of linters
< BlueMatt> mostly cause putting them in one place is clean from a "this isnt our code, you should know that" human pov
< BlueMatt> but, yea, my preference for anything where linters come up is to remove them :)
< wumpus> soo
< MarcoFalke> wumpus: Any reason to not run doxygen on it?
< wumpus> I had to change 10 places (and more) just to add one subtree
< wumpus> MarcoFalke: because it's not our code
< wumpus> I don't want general google documentation in our doxygen
< jonasschnelli> Can't attend the meeting today...
< wumpus> it's meant to document bitcoin core, not leveldb, or a crc32 library
< bitcoin-git> [bitcoin] fanquake opened pull request #17403: doc: reintegrate 0.19.0 release notes (0.19...reintegrate_relnotes_019) https://github.com/bitcoin/bitcoin/pull/17403
< MarcoFalke> ok, fine
< wumpus> do you think it would not only be clutter?
< wumpus> in any case that leaves the linters and everything
< wumpus> I think a mess now
< wumpus> it took me an hour or so to find them all
< MarcoFalke> #action Move subtrees to one place
< sipa> it seems ctaes and secp256k1 are included in doxygen, though
< wumpus> that's probably accidental
< jnewbery> an alternative would be to have an exclude.txt file in the linter directory and have all linters read from there
< wumpus> it's also easy to forget one
< jnewbery> (not saying we shouldn't change code organization structure, just agree with Marco that it shouldn't be driven by satisfying the linters)
< wumpus> yes, that's also possible
< wumpus> oh I agree
< sipa> or have a directory subtrees with all the subtrees? :)
< fanquake> another alternative would be to just remove all the linters, as they seem to take up a disproportionate amount of everyones time ?
< MarcoFalke> fanquake: Not having them also takes up time, unfortunately (as we recently saw)
< jnewbery> Concept ACK directory subtrees with subtrees
< jamesob> something something baby bathwater
< wumpus> FWIW doxygen input is already generated using automake so it'd be trivial to generate it from a central list
< wumpus> the linters are shell scripts all over the place
< BlueMatt> ignoring linters, I think its a good idea to put them all in one place, and it looks like no one materially disagreed?
< wumpus> using different ways to scan the tree
< wumpus> so that's more work...
< BlueMatt> so...next topic?
< fanquake> I have a real quick topic if there aren't any others left
< BlueMatt> i haz topic
< MarcoFalke> It is going to change all includes. Incoming merge conflicts
< wumpus> #topic rust in bitcoin core (BlueMatt)
< wumpus> MarcoFalke: yes, it's a mess to clean up a mess
< MarcoFalke> lol
< BlueMatt> one question about the rust stuff that wasn't answered in the last meeting (#16834)...in the last meeting the conclusion was "why merge if not shipped by default", so I set it as default, and this added a new question...if you dont have rustc installed, should configure fail, or just warn and not build rust.
< BlueMatt> cfields made the point that this results in silently building binaries with significantly different feature sets than release ones, which feels super strange, but I figure folks will complain if suddenly master stops building for them cause rustc is required unless you say --disable-rust. thoughts?
< gribble> https://github.com/bitcoin/bitcoin/issues/16834 | Fetch Headers over DNS by TheBlueMatt . Pull Request #16834 . bitcoin/bitcoin . GitHub
< BlueMatt> other than this it seems that pr is Ready to Go (tm)
< jamesob> seems pretty aggressive to me to require rustc by default
< wumpus> but if someone wants to fix the linters to use a central exclude list that's fine with me too ,anyhow, I really don't know what's the best solution here, but I think something needs to change
< fanquake> I don't think rustc should be a requirement to build master, I also agree that silently failing to build new feature sets feels like the wrong behaviour.
< wumpus> also I didn't really like including crc32c in our main src directory, it's kind of indirect
< wumpus> something like /src/external would be better
< sipa> why not just leave the rust stuff off by default?
< wumpus> yes, leave the rust stuff off by default, at least initially
< BlueMatt> sipa: you're three meetings late on that one :p
< sipa> yeah, haven't paid much attention to that
< sipa> but i don't see why that isn
< sipa> 't the obvious solution
< jnewbery> Sorry, I also missed the 'on by default' decision
< BlueMatt> well I mean we could also leave it off by default, but there was strong agreement to have it on by default in 0.20
< BlueMatt> I dont think there were any voices that disagreed with that
< promag> hi
< jnewbery> BlueMatt: do you have a link? Struggling to find it in my scrollback
< wumpus> I haven't seen that
< wumpus> I like to merge *some* rust code by 0.20
< BlueMatt> (built by default, not, like, making new network calls by default at runtime, that is)
< wumpus> I think it's best to have it disabled by default initally
< meshcollider> Maybe the "strong agreement" was just everyone missing the discussion :p
< jamesob> and just to clarify, there's no way that a failure in any of the rust subsystems can cascade over to the other processes, right?
< BlueMatt> (in case that was the confusion)
< wumpus> it's always possible to change a default
< wumpus> but you're going to have a merge harder time getting it merged it you want to require it
< BlueMatt> yes, jamesob, that is (to my knowledge) true.
< fanquake> #17090 some default discussion in here
< gribble> https://github.com/bitcoin/bitcoin/issues/17090 | RFC: Rust code integration . Issue #17090 . bitcoin/bitcoin . GitHub
< BlueMatt> wumpus: I personally dont want to require it, my preference was just "on if you have rustc available"
< BlueMatt> but cfields pointed out this felt....strange
< BlueMatt> cause you would just silently get things that had different feature sets from release
< wumpus> I like explicit configuration flags, but anyhow
< promag> BlueMatt: isnt that the same as with qt?
< jamesob> I think in general whenever we release major new features they should be opt-in (if possible) for a few releases to iron out kinks
< BlueMatt> you mean ./configure flags, anything written so far has to be opt-ed in at runtime
< sipa> promag: but Qt is actually built and enabled by default in release builds
< BlueMatt> promag: maybe zmq is a better example, at least with qt you just totally miss the bitcoin-qt binary
< meshcollider> why not off unless you have --enable-rust then?
< BlueMatt> jamesob: you mean opt-in, or not built-in, cause those are different
< wumpus> I'm not sure why this is such a big issue, just leave it experimental for the first release and only activated with a flag
< jamesob> BlueMatt: I mean available but disabled by default
< wumpus> this can always be chagned later
< fanquake> I feel like an explicit opt into rust stuff for 0.20.0 is ok. That would be the same as all this new android stuff. None of that is going to be anything other than explicitly opt in for a while
< promag> +1 fanquake
< wumpus> fanquake: exactly
< BlueMatt> wumpus: I'm confused, you mean on by default at runtime or built by default
< sipa> by default don't compile it in
< wumpus> it's much *easier* to merge things if they're introduced that way
< BlueMatt> same goes for fanquake...
< wumpus> disabled in the compile by default
< jnewbery> wumpus: +1
< jamesob> +1
< fanquake> BlueMatt I mean if you want to do rust things, do ./configure --enable-rust, and that's the only time it's used
< BlueMatt> lol wumpus i distinctly recall the opposite thought last meeting
< BlueMatt> but, ok, I can undo all those changes...
< BlueMatt> fanquake: lol you too...
< jnewbery> BlueMatt: no-one else remembers. I can't see any agreement in the Oct 10 meeting
< promag> BlueMatt: just to be clear, you suggest to enable if rustc is available?
< wumpus> I'm not sure having rustc in your path should change anything automatically to bitcoin configuration
< BlueMatt> yea, that was cfields' objection, which seems strange
< wumpus> but than again, I like explicit configuration options, I hate 'intelligent' software trying to second-guess me in general, so :)
< wumpus> any other topics?
< promag> I think it's fair to expect a first-time thing to be opt-in
< fanquake> I have one in regards to #15847
< gribble> https://github.com/bitcoin/bitcoin/issues/15847 | Feedback for GitHub CEO . Issue #15847 . bitcoin/bitcoin . GitHub
< wumpus> #topic github feedback (fanquake)
< fanquake> Basically just for everyone to dump any more thoughts they have into that thread, as the (in person) discussions with GitHub are happening early next week.
< wumpus> BlueMatt: let' sjust aim to get some rust code in, the default discussion is really something that can be left for later and only confuses the intial merge I think
< fanquake> There have already been some discussions in repos in GitHub in which I've started bringing up some of our suggestions.
< BlueMatt> right, ok, I'll just turn it off by default and we can have this conversation again post-merge.
< wumpus> yes, maybe in a major release or two
< promag> wumpus: I have one topic: cmd notify queue
< fanquake> Seems that some of the maintainers from other projects, which include some people from Rust core, agree with some of our concerns/issues.
< fanquake> That was all though. If you have any thoughts you don't wont to post in the issue, feel free to get in touch with me directly.
< sipa> fanquake: come say hi when you're in the bay area
< fanquake> sipa: will do
< wumpus> #topic cmd notify queue (promag)
< promag> so -walletnottify and -blocknotify spawn a thread which in turn call system()
< promag> and sometimes that can lead to some load - but that's not the issue now
< wumpus> you want an unbounded queue instead of a fork bomb :)
< bitcoin-git> [bitcoin] sipsorcery opened pull request #17404: Remove redundant class file include from test_bitcoin msvc project (master...msvc_test) https://github.com/bitcoin/bitcoin/pull/17404
< wumpus> launching processes for notifications was always a bad idea
< promag> well, that can't be changed I guess
< wumpus> definitely if they happen often enough to consider things like that
< wumpus> well if it's just every 10 minutes no one complains
< promag> while replacing with boost::process I've noticed that the pace of process spawning is shorter
< wumpus> if it's a high frequency noticication you should use another mechanism
< promag> so I wonder wdyt about adding some queue
< promag> which has de bonus of guaranteeing order
< promag> also, the advantage is to avoid command line placeholders
< wumpus> that adds latency, also people expect the thread to run free (so keeping a command running doesn't block bitcoind), so you can't guarantee much
< promag> and pass vars via env
< promag> wumpus: I think that if a command does that now I will have problems anyway
< promag> *it will
< wumpus> right now, it creates a new thread an starts the command in that
< promag> right, but if the command doesn't quit you end up with lots of threads?
< wumpus> if you want to create say, a single-threaded worker queue for notifications, that does change how it works
< wumpus> I'm sure it needs to quit some time
< promag> would you nack that?
< wumpus> I think it's a unnecessary change
< wumpus> how it is now works for the people that use the current mechanism
< wumpus> if you want more, use zmq
< wumpus> you're not going to make a system that supports high frequency notifications with process spawning
< promag> my problem is that in #13339 if I use boost::process notifications are slowly handled
< gribble> https://github.com/bitcoin/bitcoin/issues/13339 | wallet: Replace %w by wallet name in -walletnotify script by promag . Pull Request #13339 . bitcoin/bitcoin . GitHub
< wumpus> why?
< wumpus> is boost process so inefficient? what does it do differently?
< promag> my guess is that because internally it has to guard getenv/send etc
< promag> setenv
< sipa> how does a queue help?
< wumpus> promag: why?
< wumpus> you can provide an environment when you exec
< wumpus> there's no need to ever call setenv
< wumpus> in a multi-threaded process you definitely don't want to use that
< promag> at a lower level those calls aren't thread safe
< wumpus> there are thread safe calls at a low leven
< wumpus> l
< wumpus> oh, it's time
< wumpus> #endmeeting
< lightningbot> Meeting ended Thu Nov 7 20:00:28 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< sipa> #lunch
< promag> sipa: adopting a queue to guarantee order doesn't fix that, but in turn its no longer a problem :P
< wumpus> promag: man execvpe
< jnewbery> wumpus: MarcoFalke: meshcollider: 15931 I think is ready for merge. ACKs on 36b68de from ryanofsky, jnewbery, jkczyz, promag
< MarcoFalke> #15931
< promag> ill dig a little more, ty
< gribble> https://github.com/bitcoin/bitcoin/issues/15931 | Remove GetDepthInMainChain dependency on locked chain interface by ariard . Pull Request #15931 . bitcoin/bitcoin . GitHub
< meshcollider> jnewbery: sweet I'll take a look today
< jnewbery> thanks!
< wumpus> promag: sorry to be so cynical but the way you way that, the more I hear people talking about boost::process the more I start thinking that it's a bad idea to start using it
< wumpus> things like thread safe calls for process spawning have been solved in the 90's, it's not some magic new technology
< promag> The execvpe() function is a GNU extension.
< wumpus> so do you need to override the environemtn?
< wumpus> we've always passed things as arguments
< promag> wumpus: escaping issues
< wumpus> wait
< promag> and honestly do you prefer that over env vars?
< wumpus> you're talking about low-level process spawning right?
< wumpus> escaping issues are a shell thing
< jeremyrubin> Oh
< jeremyrubin> Meeting....
< wumpus> if you don't invovle the shell, so spawn a process directly, you're not going to have escaping issues
< wumpus> this is similar to python subprocess.Popen with shell=True versus shell=False
< wumpus> yes, I prefer arguments to environment variables
< ryanofsky> wumpus, on windows you have escaping issues without a shell because processes are starting with a single string, not an array of strings, and the c runtime library parses that into an argv array
< wumpus> environment variables are for global state
< wumpus> if you need to pass arguments to a process, the preferred way to do that is to use arguments
< promag> wumpus: what I usually see is if something has to launch a process with a custom configuration then env vars is the choice - honestly I don't recall seeing placeholders
< promag> like %s
< promag> anyway, I'll try to see what is slowing boost::process:spawn
< wumpus> ryanofsky: I have no idea about windows
< promag> I really dislike %w, and there's the windows "'issue'", that's why I'm trying with boost::process
< wumpus> POSIX already confuses me enough,thanks
< wumpus> let's look it up though
< promag> yup, I don't think this stuff too.. but walletnotify lacks support for multiwallet
< wumpus> which brings me back to, please use another notification mechanism
< promag> wumpus: we don't support zmq in the wallet
< wumpus> spawning processes is OS specific madness
< promag> right
< wumpus> in a sandboxed env you might not even be allowed to do that
< wumpus> promag: maybe we should!
< promag> I think that good solutions is a) some PR for longpoll by jonasschnelli b) push to some amqp
< promag> problem with a) you need a long poll for each wallet
< wumpus> so to be clear I'm not arguing to remove walletnotify etc, if the load is low enough that spawning processes is good enoguh, then it's good enough, but I don't think it's worth spending a lot of work to improve/change it, especially incompatibly or movingthings to environment variables etc
< promag> wumpus: right, I'm ok with last decision - just support %w in linux/mac
< wumpus> so why is wallet zmq not a thing?
< wumpus> why notwork on that instead?
< promag> ?\_(?)_/?
< promag> I think that was discussed before
< wumpus> okay
< promag> if the subscriber is down then you miss notifications?
< promag> I don't remember
< sipa> zmq is only useful as a "you need to poll right now" mechanism
< wumpus> I don't think spawning processes is 100% guaranteed either
< sipa> you need to poll at startup/intermittently anyway; zmq helps reduce your latency
< wumpus> it's extremely hard, maybe impossible to make a notification mechanism that is 100% loss proof
< promag> so the zmq message is (new stuff in wallet x, bye)
< wumpus> sipa: right
< wumpus> and it includes sequence numbers so you know if you missed something
< wumpus> (maybe use a block chain for your notifications *ducks)
< promag> that's so 2010..
< bitcoin-git> [bitcoin] fanquake pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/270616228bc9...2fb6140d585f
< bitcoin-git> bitcoin/master b80f7db Aaron Clauson: Remove redundant class file includes from test_bitcoin project.
< bitcoin-git> bitcoin/master 2fb6140 fanquake: Merge #17404: Remove redundant class file include from test_bitcoin msvc p...
< bitcoin-git> [bitcoin] fanquake merged pull request #17404: Remove redundant class file include from test_bitcoin msvc project (master...msvc_test) https://github.com/bitcoin/bitcoin/pull/17404
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #17405: wallet: Remove unused boost::this_thread::interruption_point (master...1911-walletNoBoost) https://github.com/bitcoin/bitcoin/pull/17405
< bitcoin-git> [bitcoin] BlockMechanic opened pull request #17406: Fix http server (master...fix-http-server) https://github.com/bitcoin/bitcoin/pull/17406
< bitcoin-git> [bitcoin] BlockMechanic closed pull request #17406: Fix http server (master...fix-http-server) https://github.com/bitcoin/bitcoin/pull/17406
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #17407: Add MempoolInstance() to get the current mempool (master...1911-txPoolOptional) https://github.com/bitcoin/bitcoin/pull/17407
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #17408: Replace assert with ASSERT (master...1911-assertASSERT) https://github.com/bitcoin/bitcoin/pull/17408
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #17408: Replace assert with ASSERT (master...1911-assertASSERT) https://github.com/bitcoin/bitcoin/pull/17408