< * luke-jr> wonders if it would help to get force-pushes announced here. otoh, might get spammy
< BlueMatt> uhhhh...hopefully there re exactly 0?
< jeremyrubin> I force push all the time :| How else do you rebase?
< BlueMatt> I figured he meant to the repo itself
< sipa> jeremyrubin: what BlueMatt said
< jeremyrubin> i think i was thrown off by luke saying spammy -- I thought we should see 0!
< wumpus> moneyball: so they have the requirement based on the age of the software apparently
< wumpus> luke-jr: afaik, force-pushes are announced here, at least the old bot did
< wumpus> there just aren't any, usually
< provoostenator> I'm able to install from the signed DMG just fine (overriding a previous installation)
< provoostenator> I suppose it's alright if Apple provides additional free malware review :-) (as long as they keep the right-click to install alternative)
< JeremyCrookshank> Any recommenced reading for Bitcoin Development?
< luke-jr> wumpus: force-pushes to PRs?
< luke-jr> ie, rebases
< bitcoin-git> [bitcoin] jonatack opened pull request #17327: test: add rpc_fundrawtransaction logging (master...rpc_fundrawtransaction-test-logging) https://github.com/bitcoin/bitcoin/pull/17327
< provoostenator> Looks like #16697 is back. I'm seeing it again on master too....
< gribble> https://github.com/bitcoin/bitcoin/issues/16697 | Unknown version bit fork activated warning . Issue #16697 . bitcoin/bitcoin . GitHub
< wumpus> luke-jr: the bot only monitors the branches on the main repository, not every user's branches
< wumpus> and it shouldn't imo
< wumpus> provoostenator: I thought we deleted that code
< wumpus> apparently not
< provoostenator> Fun fact: downloading from internet is different than copying via SSH from a Gitian machine... for macOS
< luke-jr> wumpus: it announces PR opens/closes
< luke-jr> provoostenator: what?
< wumpus> luke-jr: right, I suppose it *could* track all user's branches too, but I think that would be a bad idea, I'm not interested in rebases on 294 open PRs
< luke-jr> wumpus: it seems more interesting than DrahtBot's close/reopen Travis rebuilds ;)
< luke-jr> wumpus: not all user branches are PRs tho
< luke-jr> anyway, I just meant in terms of timely review of rebased PRs before they need rebase yet again a day later <.<
< wumpus> it would be possible to filter out drahtbot actions
< wumpus> provoostenator: windows does a similar thing, some kind of taint tracking with files, anything downloaded by the browser is suspect
< luke-jr> wget does it too, just doesn't warn you
< luke-jr> getfattr -d on your downloads
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/08e29473126d...feb1a8c03aff
< bitcoin-git> bitcoin/master 3b3b931 Carl Dong: nsis: Write to correct filename in first place
< bitcoin-git> bitcoin/master feb1a8c Wladimir J. van der Laan: Merge #17308: nsis: Write to correct filename in first place
< bitcoin-git> [bitcoin] laanwj merged pull request #17308: nsis: Write to correct filename in first place (master...2019-10-simplify-nsis-exe-rename) https://github.com/bitcoin/bitcoin/pull/17308
< JeremyCrookshank> jonatack & wumpus thank you
< JeremyCrookshank> already done two basic GUI PRs just wanting to expand knowlege
< JeremyCrookshank> will be reviewing #16432 soon
< gribble> https://github.com/bitcoin/bitcoin/issues/16432 | qt: Add privacy to the Overview page by hebasto . Pull Request #16432 . bitcoin/bitcoin . GitHub
< instagibbs> luke-jr, I force push a ton to my PRs, it would be a bit much
< luke-jr> hmm
< luke-jr> maybe we should just try manually announcing when a rebase is ready
< instagibbs> rebase is ready?
< instagibbs> meaning, "it's ok to re-review now"?
< luke-jr> right
< instagibbs> I usually manually announce in the thread, yeah
< provoostenator> I force push so much it bogs down Travis :-)
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/feb1a8c03aff...1c5e0ccabae1
< bitcoin-git> bitcoin/master 9cae3d5 practicalswift: tests: Add fuzzer initialization (hold ECCVerifyHandle)
< bitcoin-git> bitcoin/master 1c5e0cc MarcoFalke: Merge #17274: tests: Fix fuzzers eval_script and script_flags by re-adding...
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #17274: tests: Fix fuzzers eval_script and script_flags by re-adding ECCVerifyHandle dependency (master...fuzzer-initialization-follow-up) https://github.com/bitcoin/bitcoin/pull/17274
< bitcoin-git> [bitcoin] darosior opened pull request #17328: GuessVerificationProgress: cap the ratio to 1 (master...fixup_getblockchaininfo) https://github.com/bitcoin/bitcoin/pull/17328
< bitcoin-git> [bitcoin] jnewbery opened pull request #17329: linter: Strip trailing / in path for git-subtree-check (master...2019-10-subtree-path) https://github.com/bitcoin/bitcoin/pull/17329
< JeremyCrookshank> "Take pains to explain exactly what you're doing and the reasoning behind"
< JeremyCrookshank> Pain? ;)
< bitcoin-git> [bitcoin] sdaftuar opened pull request #17330: [qa] Add shrinkdebugfile=0 to regtest bitcoin.conf (master...2019-10-shrinkdebugfile0) https://github.com/bitcoin/bitcoin/pull/17330
< bitcoin-git> [bitcoin] MarcoFalke pushed 6 commits to master: https://github.com/bitcoin/bitcoin/compare/1c5e0ccabae1...100fa0a62a29
< bitcoin-git> bitcoin/master 52cf68f Russell Yanofsky: Refactor: Add GetLegacyScriptPubKeyMan helper
< bitcoin-git> bitcoin/master 628d11b Russell Yanofsky: Add back mistakenly removed AssertLockHeld
< bitcoin-git> bitcoin/master 2632b1f Russell Yanofsky: doc: Clarify WalletStorage / Wallet relation
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #17300: LegacyScriptPubKeyMan code cleanups (master...pr/keyman-cleanup) https://github.com/bitcoin/bitcoin/pull/17300
< bitcoin-git> [bitcoin] fanquake pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/100fa0a62a29...811c381ab650
< bitcoin-git> bitcoin/master 60582d6 John Newbery: [linter] Strip trailing / in path for git-subtree-check
< bitcoin-git> bitcoin/master 811c381 fanquake: Merge #17329: linter: Strip trailing / in path for git-subtree-check
< bitcoin-git> [bitcoin] fanquake merged pull request #17329: linter: Strip trailing / in path for git-subtree-check (master...2019-10-subtree-path) https://github.com/bitcoin/bitcoin/pull/17329
< bitcoin-git> [bitcoin] achow101 opened pull request #17331: Use effective values throughout coin selection (master...effective-value) https://github.com/bitcoin/bitcoin/pull/17331
< jeremyrubin> meeting?
< warren> here
< achow101> meeting?
< wumpus> #startmeeting
< lightningbot> Meeting started Thu Oct 31 19:00:21 2019 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
< jnewbery> hi
< warren> hi
< kanzure> hi
< achow101> hi
< amiti> hi
< moneyball> hi
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/811c381ab650...222b7d0ca795
< bitcoin-git> bitcoin/master c5377ff Suhas Daftuar: [qa] Add shrinkdebugfile=0 to regtest bitcoin.conf
< bitcoin-git> bitcoin/master 222b7d0 MarcoFalke: Merge #17330: test: Add shrinkdebugfile=0 to regtest bitcoin.conf
< jeremyrubin> im unsure about DST
< jeremyrubin> Oh
< sipa> hi
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #17330: test: Add shrinkdebugfile=0 to regtest bitcoin.conf (master...2019-10-shrinkdebugfile0) https://github.com/bitcoin/bitcoin/pull/17330
< provoostenator> hi
< wumpus> yes, DST in europe changed
< wumpus> it's an hour earlier here
< meshcollider> wumpus: don't forget to ping everyone :p
< meshcollider> hi
< fanquake> hi
< wumpus> four proposed topics in https://gist.github.com/moneyball/071d608fdae217c2a6d7c35955881d8a : "moneyball: follow-up on travis and #16148" "umpus: close Boost -> C++11 migration project for now" "proposed by dongcarl: MacOS, notarization https://github.com/bitcoin/bitcoin/issues/15774" "jeremyrubin: epoch mempool"
< gribble> https://github.com/bitcoin/bitcoin/issues/16148 | Travis timeouts . Issue #16148 . bitcoin/bitcoin . GitHub
< bitcoin-git> [bitcoin] sdaftuar opened pull request #17332: [p2p] Proof-of-concept: Improve DoS-resistance to low-work headers chains (master...2019-10-no-checkpoints-cleanedup) https://github.com/bitcoin/bitcoin/pull/17332
< digi_james> Hi
< wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb moneyball kvaciral ariard digi_james amiti fjahr
< wumpus> #topic High priority for review
< wumpus> currently 7 blockers, 7 chasing concept ACK
< BlueMatt> plz2add #16974
< gribble> https://github.com/bitcoin/bitcoin/issues/16974 | Walk pindexBestHeader back to ChainActive().Tip() if it is invalid by TheBlueMatt . Pull Request #16974 . bitcoin/bitcoin . GitHub
< wumpus> BlueMatt: added (to blocker I guess?)
< BlueMatt> yea
< BlueMatt> it blocks post-headers-over-dns rust progress (since I want to lean heavily on "I have a better header than blocks, but my peers dont have blocks" as a criteria for searching for alternate block download methods)
< wumpus> yes, makes sense
< wumpus> nothing else too add/remove?
< wumpus> #topic close Boost -> C++11 migration project for now (wumpus)
< warren> too much change required?
< wumpus> so it looks like all the low-hanging fruit for replacing boost has been picked now
< wumpus> what remains is hard to replace with C++11 (results in very verbose code, locale dependent legacy C, or introduces harder to maintain platform-specific code)
< jeremyrubin> It's basically just multiindex and boost::thread now right?
< wumpus> it's kind of a time wasting trap for developers now (regarding PRs like #17245)
< gribble> https://github.com/bitcoin/bitcoin/issues/17245 | wallet: Remove Boost from DecodeDumpTime by elichai . Pull Request #17245 . bitcoin/bitcoin . GitHub
< wumpus> nah, people stumble over the sleep and date/time handling stuff every time
< wumpus> #17307 might still be worthwhile
< gribble> https://github.com/bitcoin/bitcoin/issues/17307 | Stop using `boost::thread_group` . Issue #17307 . bitcoin/bitcoin . GitHub
< jeremyrubin> What about just checking in those specific copies of boost library code
< sipa> after 17307, won't removing boost::threa dbe a lot easier?
< jeremyrubin> Or are they too large/dependent on the rest of boost types
< wumpus> but it needs to be focused, we don't want to close the same PRs time after time that don't really make it
< wumpus> I think having that project open sends the wrong messag
< wumpus> we can't replace boost right now
< jeremyrubin> 17307, having worked on replacing thread_group in the checkqueue before, scares me a bit
< sipa> agree there
< wumpus> there's no need to replace, say, small string handling functions with our own impelentation, before we have a vision to replace the rest
< sipa> right; until we have a reasonable way to remove multiindex (which i'm not sure will ever happen), getting rid of boost entirely is not really useful
< dongcarl> Just so it's out there... we can maybe run bcp at some point when there's only one or two things left https://www.boost.org/doc/libs/1_71_0/tools/bcp/doc/html/index.html
< jeremyrubin> dongcarl: ++
< sipa> i do think there are individual improvements possible that dkn't go all the way, like removing boost::thread
< jnewbery> I agree that if it's not a priority then the project should be closed. It can always be re-opened later if necessary.
< wumpus> but in any case it doesn't seem like it needs a big coordinated project anymore
< sipa> how many non-headers-only boost libs are we still using?
< sipa> wumpus: agree
< jnewbery> (leaving a comment in the project description explaining why it's been closed)
< wumpus> when C++17 is allowed, it can be reopened
< fanquake> Guess #13751 can be closed with the same rationale as well then.
< gribble> https://github.com/bitcoin/bitcoin/issues/13751 | Utils and libraries: Drops the boost/algorithm/string/split.hpp dependency by l2a5b1 . Pull Request #13751 . bitcoin/bitcoin . GitHub
< wumpus> fanquake: yes
< wumpus> there's not really a reason to do that right now
< warren> gitian binaries static link the libstdc++ so distro support won't change when C++17 happens right?
< dongcarl> warren: Yes
< wumpus> warren: correct, assumning we can keep the same GLIBC req
< wumpus> (runtime)
< wumpus> which I don't see why not
< wumpus> statically linking using musl libc is another thing that might be considered in the future
< wumpus> but that's a whole different topic
< dongcarl> should be able to keep it as long as we 1. stick to Bionic 2. write new hooks, ick, or 3. use guix
< cfields> Bionic?
< dongcarl> cfields: bionic has 1.27, which is what we've written hooks to support up to
< dongcarl> glibc 1.27 that is
< sipa> i read "books" instead of hooks. was confused
< bitcoin-git> [bitcoin] fanquake closed pull request #13751: Utils and libraries: Drops the boost/algorithm/string/split.hpp dependency (master...patch/remove_boost_split) https://github.com/bitcoin/bitcoin/pull/13751
< wumpus> hehe
< jnewbery> Shall I go ahead and update the project description and then close?
< wumpus> jnewbery: yes, thanks
< jnewbery> ok done
< wumpus> seems no one strongly disagrees
< wumpus> #topic MacOS, notarization (dongcarl)
< jnewbery> updated description is visible here: https://github.com/bitcoin/bitcoin/projects?query=is%3Aclosed
< dongcarl> Right, here
< wumpus> #15774
< gribble> https://github.com/bitcoin/bitcoin/issues/15774 | macOS App Notarization . Issue #15774 . bitcoin/bitcoin . GitHub
< wumpus> jnewbery: great!
< dongcarl> Mostly just to make people aware of what's going to happen
< dongcarl> We're going to have a step in the release process that can only happen on a mac
< cfields> dongcarl: I think we should consider rejecting Apple's new requirement.
< wumpus> discntinuing MacOS binaries?
< dongcarl> cfields: that's something I've thought about too
< provoostenator> It depends on what the step is
< * dongcarl> typing gimme a sec
< cfields> I've been putting something together, probably best not to thought-dump here.
< provoostenator> As long as it's easy to review the difference on a non-Mac I don't mind too much.
< wumpus> htey make it harder and harder to distribute binaries for open source software
< luke-jr> [19:16:33] <warren> gitian binaries static link the libstdc++ so distro support won't change when C++17 happens right? <-- gitian binaries don't matter in this regard
< luke-jr> distro support is for NATIVE compiles
< luke-jr> (also, gitian binaries *don't* work on all distros)
< dongcarl> After we submit the app for notarization it is actually not NEEDED to staple it to the binary
< dongcarl> Apparently GateKeeper will query the Apple servers
< wumpus> luke-jr: that was last topic, but yes
< dongcarl> Which is convenient, but gives them A LOT OF CONTROL
< provoostenator> I'm not a huge fan of Apple getting a call from all our useres
< luke-jr> oh, I missed the #topic line
< provoostenator> So better to staple if it's not too difficult
< cfields> the bigger issue is that it makes Apple the final arbiter of what people can run. In a potential emergency fork/uasf scenario, this puts them in charge. I think it would be unwise for us to participate in that.
< luke-jr> wumpus: is Apple dropping support for unsigned binaries entirely?
< cfields> (final arbiter once we start down the path, that is)
< wumpus> yes, they want the same control as they have on iOS
< provoostenator> What changes when we go down the path?
< luke-jr> ugh
< moneyball> luke-jr: no you can still install but you have to "right click open"
< luke-jr> moneyball: right now, yes; but after these new changes?
< cfields> I think it's a mistake to engage that, because we can't go back. And if they refuse to sign a soft-fork, presumably there'd be nothing we could do.
< provoostenator> I don't understand why we can't go back
< luke-jr> cfields: we could timebomb..
< provoostenator> Do they change the installation rules once you notarize once?
< luke-jr> Knots already expires 1-2 years after the release
< dongcarl> I'm less worried about them refusing to sign
< dongcarl> I'm more worried about them signing: we've audited this binary and it's malicious
< moneyball> luke-jr: I run MacOS Catalina, am able to install 0.18 fine, and 0.19 RC3 I can install but need to "right click Open"
< cfields> provoostenator: if we engage once, then it would be a regression not to continue.
< dongcarl> That's a bad message to users
< provoostenator> cfields: arguably we're just kicking that regression down the road
< moneyball> If we make no changes, I think at minimum we need to communicate on the download site that one must "right click Open" otherwise many users will be perplexed as I was
< wumpus> it's not a regression if we never start supporting it
< provoostenator> Because Catalina users will encounter the regression (right click to install) now
< provoostenator> Which we can delay
< luke-jr> As long as users can run binaries w/o the notary, that sounds preferable
< wumpus> I think it's a good point, for a project like us, to be principled about it
< cfields> provoostenator: yes,( unpopular opinion time...) which is why it's worth considering dropping MacOs
< luke-jr> cfields: why drop entirely?
< cfields> I'm not saying that we should. I'm saying it's worth a discussion.
< wumpus> only when they *require* this
< wumpus> not right now
< luke-jr> can't our DMG just tell them right click etc in the folde rview?
< warren> are there other reasons aside from this?
< wumpus> but it's a possiblility in the future
< jeremyrubin> is there no more self-signing?
< wumpus> if it really becomes a closed platform
< sipa> cfields: stop supporting for release binarier, or stop supporting the platform? (i know the distinction is kinda vague for us, but we can still make sure our code compiles and runs fine, even if we're not providing binaries)
< jeremyrubin> Can we have a binary which we get apple to sign which self-signs our binary?
< wumpus> sipa: you can still build your own
< wumpus> sipa: using homebrew
< achow101> What changed between 0.18.x and 0.19 that 0.19 now triggers the warning but 0.18.x doesn't?
< jeremyrubin> Then apple would only have to sign it once, and then that can self-sign future cores
< wumpus> achow101: nothing in our code
< cfields> sipa: yes, thanks for making the distinction. The first would be far less problematic.
< dongcarl> Here's the risk that I see: someone takes our codesigned binaries, gives it to Apple, Apple maybe mistakingly decides that the binary is malicious, and now everyone's GateKeeper will ask Apple about that codesigned binary, which will give them a "THIS IS MALICIOUS" popup on double-click
< achow101> wumpus: moneyball said that 0.18.x installs fine, but 0.19 doesn't, on the same system
< wumpus> achow101: only because iti's more recent I think
< sipa> cfields: devil's advocate... is this very different from say snap controlling our snap releases? sure, snap is not the only way you run bitcoind on ubuntu, but depending on the user's expertise there may not be that much difference in how much control those entities have over what consensus code gets adopted
< cfields> dongcarl: yes, and that's inevitible. Because we ARE bitcoin core. You know, the thing they're scanning their binaries for :)
< fanquake> I'd assume he'd already been running 0.18
< moneyball> fanquake: me? as an experiment i installed 0.18 yesterday to compare to the experience installing 0.19
< achow101> I think it would be worthwhile to investigate why that happens and see if we are able to avoid the warning without doing any apple special stuff
< luke-jr> [19:27:31] <warren> are there other reasons aside from this? <-- macOS only works on backdoored hardware; macOS is proprietary; etc
< fanquake> moneyball yea. I don't think the reinstalling 0.18 would make a difference though.
< cfields> achow101: yes, agree.
< wumpus> even google isn't aiming for that kind of total control of android
< fanquake> Otherwise suddenly everyone who updated to 10.15's software would start breaking.
< wumpus> sipa: your comparison would hold for the apple app store, not for things installed outside it
< fanquake> I'd assume the new checks only apply to "new" binaries you are trying to run.
< cfields> Admittedly I don't have enough info on this. But what I've read has really rubbed me the wrong way. So I'll shut up now and chime in on the issue instead :)
< fanquake> So if you'd already been running 0.18 previously, I don't think reinstalling it is going to cause an issue.
< sipa> wumpus: fair point; but even on android you need to explicitly enable non-play store apk sources; that seems fairly similar to the right click "open anyway"
< fanquake> cfields: I assume your opinion is that we aren't shipping a macOS binary for 0.19 then? Assuming we're already at rc3?
< cfields> sipa: I'll have to think about that. I think it's different.
< provoostenator> I don't think it's very effective for a small (in terms of macOS user base) project to make a stance at this stage.
< wumpus> sipa: yes; I definiely don't think we should stop with MacOS binaries as long as that's possible
< dongcarl> Seems this is stirring up a lot of conversation, perhaps we can come back to this as the last topic?
< cfields> provoostenator: then I suppose we'd need to involve more projects :)
< ryanofsky> another danger of us not providing binaries is that someone else starts submitting and providing them, like happened with the snap store
< sipa> cfields: as far as ideology goes, supporting osx release binaries without participating in the gatekeeper stuff perhaps has more impact than dropping support entirely
< cfields> fanquake: Nah, for 0.19 I think we should just build/sign as before.
< jeremyrubin> ryanofsky: +1
< provoostenator> cfields: well, for starters it could be useful to get a bunch of projects together and try to get a physical meeting with Apple
< wumpus> provoostenator: I'm not really interested in having it being effective as a political strategy in general
< achow101> I can (probably) find a mac that doesn't have core installed and try
< provoostenator> They might be open to actually fixing this.
< wumpus> provoostenator: it would be a bad idea for bitcoin, so for bitcoin we should reject it
< fanquake> cfields: with just a warning / opening instructions on the website / in the dmg?
< sipa> provoostenator: i very much doubt that
< fanquake> ^
< sipa> provoostenator: i suspect that from their perspective, the fact that people can run unvetted binaries is a historical legacy that needs to be fixed :)
< provoostenator> Well if the answer is "we don't want to fix this, we like a gated community" then that's a nice quote to start a boycott.
< ryanofsky> dongcarl, if we do decide to "staple" the apple signature, is it still easily possible for a user to verify the executed code was built reproducibly & signed by us?
< provoostenator> ^ that's the most important question imo
< cfields> ryanofsky: that's already how it operates. We "staple" on the detached sig. It's easily auditable.
< jeremyrubin> Odd question: if we ship a binary + a VM, can we just run it in the VM always?
< wumpus> I'm not going to upload any binaries that aren't possible to verify in a distributed way
< cfields> I assume it'd be the same here. If not, surely we could write a tool to handle it.
< ryanofsky> ok, that's good at least
< achow101> is it possible to "staple" without a mac?
< dongcarl> not possible to "staple" without a mac right now
< fanquake> You'd need Xcode and associated tools
< cfields> It's not possible to do the detached sigs without a mac either.. the tooling is our own.
< sipa> is it possible to undo the stapling on non-mac?
< cfields> Er, let me rephrase...
< cfields> There's no supported way sign on non-mac as we do now, but that didn't stop us from hacking something together :)
< achow101> someone would have to reverse engineer how it's "stapled" and write a tool for it for other platforms
< cfields> So I assume we could do something similar again. Might turn into a 3-stage gitian build, though :(
< dongcarl> what cfields said
< fanquake> Liking something we can take away from this discussion: https://trac.torproject.org/projects/tor/ticket/30126
< fanquake> *likely
< sipa> cfields: what if instead of doing the stapling in the 2nd/3rd gitian stage, we write a tool that can compare a stapled binary against a deterministic one?
< sipa> that could even remove the need for the 2nd stage we have now
< wumpus> cfields: adding a slight delay is usually not a problem
< dongcarl> sipa: A notarization strip tool?
< sipa> dongcarl: yes
< cfields> sipa: I don't think it's possible to be deterministic because of the timestamping, but maybe I'm misunderstanding.
< sipa> let me start over
< dongcarl> cfields: He's talking about our existing deterministic codesigned binary
< sipa> do we actually care that the codesigning/timestamping/notarizing step is deterministic?
< wumpus> yes
< sipa> the point of determinism is showing that the shipped binaries match our source code
< wumpus> at least the attach-signature step
< warren> presumably the timestamp is part of the stapled part, while the executable part could be verified after stapling?
< wumpus> the signing itself doesn't need to be determinsitic, it's only done once by one person
< sipa> if you could *strip* the codesigning from a binary, and compare the result of that with the deterministic unsigned result, don't we achieve the same thing?
< sipa> in that case the notarizing/codesigning could be done entirely outside of gitian, removing the 2 phase process we have now
< sipa> that of course only works if the stripping tool is easily auditable
< cfields> sipa: I see. You'd have to trust the stripping tool and be able to audit the diff, but that could work.
< sipa> right, it depends on how complicated the stripping is
< wumpus> so instead of uploading my own gitian results, I'd have to upload the binary someone else sends me
< cfields> sipa: IIRC there's a reason we didn't do it that way. Can't seem to remember though, maybe not.
< dongcarl> how many people do the codesigning right now?
< sipa> wumpus: you'd upload the codesigned binary, but the gitian results contain the non-codesigned hash
< sipa> the comparison tool strips the codesigning, and compares with gitian
< achow101> dongcarl: 2. cfields for windows, jonasschnelli for mac
< dongcarl> right, if it's just one person for each platform then perhaps stripping is not too bad?
< cfields> sipa: ah, at the time, it might've just been about the tooling. Easier to add than remove. That likely isn't an issue anymore though, there's better tooling now.
< wumpus> at least now it's possible to look up the sha256 of the binaries from the SHA256SUMS.asc in gitian, and see they match, this would add an extra step
< wumpus> I don't like it but ok...
< sipa> wumpus: agreed, it's annoying; but needing to comply with ever-increasing notarization/codesigning requirements is also annoying
< sipa> it's just an idea; no need to decide anything now
< wumpus> this wouoldn't avoid that right?
< fanquake> Apparently we wont actually need the secure timestamp or hardened runtime until January next year
< provoostenator> You can still put the sha of the signed binary in the gitian result, along with the sha of the stripped version.
< cfields> Bitcoin 0.20 when? :p
< wumpus> why even bother with bitcoincore.org uploads anymore, if you go through all these hoops, can't you put it on the app store directly?
< achow101> that's in two months...
< dongcarl> wumpus: this would avoid us having to reverse engineer each notarization/codesigning step and writing a tool to do it under our reproducible builds process
< jeremyrubin> Here's a reductionist viewpoint: It's not worth jumping through too many hoops for apple users, because ultimately if the person shipping your kernel decides you don't want them running bitcoin, or a version of bitcoin, you're already, to put it lightly, fucked
< fanquake> achow101 at least it means a 0.19.0 release is simpler...
< fanquake> and gives us time to figure out future tooling
< jeremyrubin> So this is maybe a problem we can't really solve for OSX users, short of giving them a liberated computer
< sipa> back to short-term issue: are our processes already using the secure timestamp and/or hardened runtime?
< provoostenator> wumpus: app stores come with additional problems though, like auto-update
< sipa> or how hard is it to do so
< wumpus> let's go to the next issue, already spent too much time on apple junk
< wumpus> two topics left and 10 minutes
< sipa> ok
< wumpus> #topic follow-up on travis and #16148 (moneyball)
< gribble> https://github.com/bitcoin/bitcoin/issues/16148 | Travis timeouts . Issue #16148 . bitcoin/bitcoin . GitHub
< wumpus> moneyball?
< moneyball> hi
< moneyball> my name is on the topic but it is from 1 month ago
< instagibbs> moneyball, don't timeout in 10 minutes ;)
< moneyball> when we discussed the same topic and said punt 1 month
< wumpus> oh
< moneyball> since then the PR was closed
< moneyball> maybe MarcoFalke can discuss that?
< moneyball> also we were wondering the status of jonasschnelli's related project
< wumpus> ok, might be better to move to jeremyrubin's topic for now then
< moneyball> sure
< wumpus> #topic epoch mempool (jeremyrubin)
< jeremyrubin> Hi!
< jeremyrubin> So Epoch Mempool is a relatively big upgrade to the way the mempool tracks relatives state
< jeremyrubin> We replace a ton of std::sets with epoch tracking for touching txiters
< sipa> yay
< jeremyrubin> This is done because we have restrictive limits on the number of descendants, which is a problem for protocols like lightning or OP_STB
< jeremyrubin> The PR is just a start, but it's a good checkpoint on improvement progress
< jeremyrubin> (I will have follow ups that are further improvements)
< sipa> it's clear to me that that approach in theory at least should be possible and improve performance; i haven't looked at the code
< jeremyrubin> But the critical question is: what standard of "evidence" do we want to see to be comfortable with such an improvement not introducing regressions in performance?
< instagibbs> it's also a problem for services due to simple transaction pinning
< wumpus> is mempool performance a bottleneck?
< sipa> wumpus: for increasing the mempool limits, it is
< sipa> (package size, ancestor depth, ...)
< wumpus> okay
< instagibbs> sipa, is there a good writeup of "all" the reasons for the limits? There are 1 or 2 on suhas' writeup on the wiki, but likely not exhaustive of collective wisdom
< sipa> instagibbs: not that i know
< ariard> is this a change only in the way we traverse mempool sets or also in the way we compute entry feerate?
< jeremyrubin> Traversal only as of now
< jeremyrubin> The notes aren't really fully up-to-date either
< sipa> instagibbs: and sdaftuar likely knows better than i do
< instagibbs> I think I've hit up sdaftuar for all he currently has written done(which is better than 0!)
< ariard> hmmmm did you spend time to think how we compute feerate? improving this point first could ease traversal
< jeremyrubin> Feerate isn't really the chief issue here IMO
< sipa> ariard: what do you mean by compute feerate?
< jeremyrubin> But it does cause a lot of traversal-ing
< ariard> tracking ancestor/descendant size/fees
< instagibbs> the modified feerate you mean
< jeremyrubin> In either case, the feerate stuff is easier to deal with as it (iirc) ends up being O(N)-ish if you traverse in the correct order.
< jeremyrubin> But the speed of those traversals you have to pay on adding a txn or removing one
< ariard> yes nModifiesFees
< wumpus> let's wrap up please, it's almost time to close the meeting
< jeremyrubin> Ok
< jeremyrubin> I guess please review #17268
< gribble> https://github.com/bitcoin/bitcoin/issues/17268 | Epoch Mempool by JeremyRubin . Pull Request #17268 . bitcoin/bitcoin . GitHub
< wumpus> #action review #17268
< luke-jr> jeremyrubin: 1) what is "epoch tracking" ?
< gribble> https://github.com/bitcoin/bitcoin/issues/17268 | Epoch Mempool by JeremyRubin . Pull Request #17268 . bitcoin/bitcoin . GitHub
< wumpus> #endmeeting
< lightningbot> Meeting ended Thu Oct 31 20:01:21 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< instagibbs> new benchmark also https://github.com/bitcoin/bitcoin/pull/17292
< sipa> jeremyrubin: it may make sense to give a 1-sentence summary :)
< wumpus> cfields: I'll post the proposed 0.20 schedule soon, was waiting for -final, but seems we're almost there anyhow
< jeremyrubin> We basically just keep track of the last time we looked at a TxMemPoolEntry
< jeremyrubin> And if we've looked more recently than a given time, we ignore looking again.
< cfields> wumpus: oh, that was a joke. I was saying we should put out 0.20 before January so we have another release under our belt before Apple decides it doesn't like our dmgs :)
< jeremyrubin> Many of the algorithms in the mempool can be simplified greatly with this tactic
< nickler> #proposedmeetingtopic security email address for bitcoin-core/secp256k1 (https://github.com/bitcoin-core/secp256k1/pull/679)
< jeremyrubin> luke-jr: does that make sense?
< luke-jr> jeremyrubin: as for standards, does it hurt performance when using policies other than Core's feerate-only one?
< luke-jr> jeremyrubin: as for standards, does it hurt performance when using policies other than Core's feerate-only one?
< luke-jr> jeremyrubin: eg, I would want someone to evalulate if it makes priority-based mining any slower or harder to implemented
< jeremyrubin> luke-jr: Unless you have something odd, it should have no impact
< sipa> luke-jr: that's completely orthogonal; it just speeds up all recursive algorithms over transactions
< jeremyrubin> * negative impact
< sipa> am i right: (a) there is a global epoch counter in the overall mempool; (b) every time we "iterate" in a recursive algorithm through a transaction, that transaction's timestamp is set to the mempool epoch and the epoch is incremented (c) when iterating, you can ignore any transaction whose timestamp is later than the epoch of when you started iterating; (d) you make sure there are no two recursive
< sipa> algorithms simultaneously updating the...
< sipa> timestamps
< jeremyrubin> yes
< jeremyrubin> Although there are specific cases where the recursive both updating is desirable
< jeremyrubin> It's not used here
< jeremyrubin> This also opens us up to really fancy pants stuff down the line like parallelized mempool traversals
< jeremyrubin> Because the epochs are replacing data structures like std::set
< jeremyrubin> sipa: but yeah your base understanding is spot on
< jeremyrubin> THe other benefit is that we can now accumulate the results into better data structures, e.g., vec instead of set
< luke-jr> jeremyrubin: not really, but I'm having some connection issues, so I may have missed parts
< luke-jr> nickler: I wonder if it would make sense to simply have different PGP targets - it can be helpful to know a report was made even if you're not privy to the nature of it
< luke-jr> eg, if Wladimir is preparing a final release when a security issue gets reported, he knows to check with sipa or someone else if it should get delayed
< jeremyrubin> luke-jr: just read the pr if you haven't? It's like +300loc -200loc. It should have 0 impact on priority mining
< jeremyrubin> You can look at just like the first couple commits, and then you'll get the gist
< luke-jr> jeremyrubin: I'll have to re-read the log later when my connection is better
< luke-jr> or the PR
< luke-jr> k
< jeremyrubin> Just look at those two; that's the basic idea and the rest of the PR is just refinements/applications of that principal
< sipa> principle?
< jeremyrubin> yeah my bad
< nickler> luke-jr: good point. It seems like you're suggesting to use security@bitcoincore.org then? That has the downside of potentially (depending how it's actually set up) reaching more people when the report is unencrypted.
< nickler> Alternatively it's the responsibility of the secp256k1-security@bitcoincore.org people to warn Wladimir before making a release
< jeremyrubin> So another topic; can we make the benchmarking framework support asymptotic testing too?
< jeremyrubin> Or add a new one?
< jeremyrubin> E.g., it would be nice to be able to run an algorithm 10 times with inputs that are powers of 10
< jeremyrubin> And collect the outputs.
< jeremyrubin> But this is incompatible with the current framework, which has a goal of having the test take a second or something
< luke-jr> nickler: dunno, just a thought; also, would they warn me or other affected software too?
< luke-jr> probably a broader discussion of security handlign is desirable (to avoid a repeat of 17144, fix the "stuff never gets disclosed" issue, etc)
< nickler> I guess that depends and having a defined process could be valuable. Also, making libsecp releases and asking people to update regularly would help. But for now just a contact address would already be a simple and effective improvement over the current situation.
< bitcoin-git> [bitcoin] TheBlueMatt opened pull request #17335: Add test for syncing blocks generated after invalidateblock. (master...2019-10-invalid-mine-test) https://github.com/bitcoin/bitcoin/pull/17335
< bitcoin-git> [bitcoin] Rjected opened pull request #17336: scripts: search for first block file for linearize-data with some block files pruned (master...linearize) https://github.com/bitcoin/bitcoin/pull/17336