< phantomcircuit> with the fuzzing data in qa-assets do i just want to submit all of the files libFuzz added?
< sipa> yeah
< phantomcircuit> sipa, so just like git add . in qa-assets and submit a pr?
< sipa> yeah, is it a lot?
< sipa> you may want to minimize it first
< phantomcircuit> sipa, it's more than doubled the process_message directory
< phantomcircuit> that's why im asking lol
< sipa> how many files?
< sipa> how long did you run it lol
< phantomcircuit> sipa, not very long but i have uh lots of cores
< sipa> in cpu years then
< phantomcircuit> 234:54.14 * 32
< phantomcircuit> i think that's minutes seconds milliseconds?
< sipa> oh, that's not much
< phantomcircuit> yeah but i have 22k process_message files vs the 8k that was there before
< phantomcircuit> im reading the libFuzzer docs but i haven't gotten to compacting them yet
< sipa> did you use the test/fuzz/test_runner.py script?
< bitcoin-git> [bitcoin] fanquake closed pull request #16169: omnilayerDevelop88 (0.18...develop) https://github.com/bitcoin/bitcoin/pull/16169
< phantomcircuit> sipa, no
< sipa> fanquake: wow where did you find that?
< phantomcircuit> sipa, 2019
< fanquake> sipa: was looking at PRs tagged as backports, and that was one. 2nd PR like that recently that has just "appeared "
< sipa> phantomcircuit: so, create a new directory $NEWDIR, fill it with just the qa-assets' seeds for the process_message fuzzer, and then run FUZZ=process_message ./src/test/fuzz/fuzz -merge=1 -use_value_profile=1 $NEWDIR $DIRWITHYOURNEWSEEDS
< luke-jr> >_<
< fanquake> I opened an issue with GitHub ~5 days ago asking why it was happening. No reply from them yet.
< phantomcircuit> sipa, hmm so by default this is just putting the new test cases into the same directory, copying the qa-assets seeds is to prevent them from being deemed duplicates right (i saw your comment on the github issue about merging)
< phantomcircuit> i'd probably need to have a separate copy of them for that to really work i guess (although through git magic i guess i do)
< sipa> phantomcircuit: when fuzzing, did you start with the existing qa-assets, and fuzzed on top, or did you start from scratch?
< phantomcircuit> sipa, i started with the existing qa-assets
< sipa> ok
< phantomcircuit> sipa, having clearly not read enough about this, is the the sanest way to use cpu time, or should i use new seeds and merge them into the corpus?
< phantomcircuit> or should i copy the corpus, fuzz on the corpus and then merge back
< sipa> i'd do the last
< phantomcircuit> sipa, btw test_runner.py runs each test just once by default
< phantomcircuit> which i guess makes sense for actually running them
< phantomcircuit> but for fuzzing isn't super useful i think
< phantomcircuit> oh i see --generate
< phantomcircuit> sipa, ok this is working nicely i think
< phantomcircuit> sipa, ok it takes a very long time to do the merge lol
< sipa> phantomcircuit: it'll run every seed from both inputs once to determine their coverage
< phantomcircuit> sipa, i also have a bunch of 'slow-unit-' files
< phantomcircuit> sipa, interestingly some of the tests run through 100,000 runs really fast and others do not, i guess that makes sense though
< bitcoin-git> [bitcoin] hebasto opened pull request #20734: build: Make platform-specific targets available for proper platform builds only (master...201221-targets) https://github.com/bitcoin/bitcoin/pull/20734
< jonasschnelli> achow101: for the codesign test,... should I set the env var CODESIGN_ALLOCATE to the binary we have in the tar.gz (the oss one)?
< achow101> jonasschnelli: yeah
< sipa> the same codesign_allocate as will be used to attach the sig
< sipa> achow101: would it make sense for your tool to also do the attaching?
< achow101> sipa: yes, I plan on adding detaching and attaching options to it
< achow101> just need to figure out how to format the detached sig
< sipa> cool!
< sipa> i expect that will be easy compared to everything else you've done
< achow101> yes, it's mostly just a design decision
< achow101> and whether we want to maintain compatibility with the current process
< sipa> achow101: the actual codesign tool also has a detached format
< achow101> sipa: it can't be attached to a binary
< achow101> although maybe with some modifications it could
< sipa> ah
< achow101> the main problem is the separate CodeResources file. but codesign's detach option embeds that in the signature. I'll have to see if it's possible to do that in the embedded signature so that we don't have to deal with multiple files
< sipa> jonasschnelli: if you're going to try with rc3, use the codesign tool from the gitian build (without my recent patch)
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/f1dbf92ff047...e9efb64a0772
< bitcoin-git> bitcoin/master d8b9cec Patrick Strateman: inline non-member functions with body in fuzzing headers
< bitcoin-git> bitcoin/master e9efb64 MarcoFalke: Merge #20733: inline non-member functions with body in fuzzing headers
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #20733: inline non-member functions with body in fuzzing headers (master...2020-12-20-fuzz-inline) https://github.com/bitcoin/bitcoin/pull/20733
< jonasschnelli> sipa: ? ... I'm going to use achow101 codesign tool,.. not?
< jonasschnelli> I'll use the codesign_allocate tools from the rc3 (non modified)
< jonasschnelli> achow101: how should I modify the sign command?
< jonasschnelli> ${CODESIGN} -f --file-list ${TEMPLIST} "$@" "${BUNDLE}" (will not work)
< sipa> jonasschnelli: sorry, codesign_allocate tool
< jonasschnelli> sipa: yes. I'm using the non modified rc3 version
< achow101> jonasschnelli: are you up to date? I pushed some commits earlier today that added -f --file-list options so it should work
< jonasschnelli> achow101: no. I'm not up to date then.
< jonasschnelli> achow101: I'm at 56e5493,... ./detached-sig-create.sh -s "Bitcoin" gives me "codesign.py: error: argument command: invalid choice: 'sign.temp/signatures.txt' (choose from 'verify', 'sign', 'dump')"
< achow101> oh yeah
< achow101> ${CODESIGN} sign -f --file-list ${TEMPLIST} "$@" "${BUNDLE}"
< achow101> should work
< jonasschnelli> achow101: codesign.py: error: unrecognized arguments: -s
< jonasschnelli> set -x tell me: /Users/jonasschnelli/Documents/bitcoin/signapple/codesign.py sign -f --file-list sign.temp/signatures.txt -s Bitcoin dist/Bitcoin-Qt.app
< jonasschnelli> achow101: how do I pass the identity to your tool?
< achow101> ah, so it can't fetch the cert from MacOS's keychain. you need to export it as a .p12 and give the path, no -s
< achow101> there's instructions in docs/certificates.md
< jonasschnelli> -h is out of date. :)
< * jonasschnelli> reading...
< achow101> updating documentation is not a strong suit
< jonasschnelli> achow101: after entering the .p12 password (your tool) I'm getting FileNotFoundError: [Errno 2] No such file or directory: '/Users/jonasschnelli/Desktop/bitcoin-0.21.0rc3-osx-unsigned/dist/Bitcoin-Qt.app/Contents/_CodeSignature/CodeResources'
< jonasschnelli> My execution is: ${CODESIGN} sign /Users/jonasschnelli/Desktop/mycert.p12 -f --file-list ${TEMPLIST} "$@" "${BUNDLE}"
< jonasschnelli> (I'll remove the -s and removed the help/parameter-check in detached-sig-create)
< achow101> does that file exist after the tool fails?
< jonasschnelli> achow101: nope... there is no _CodeSignature folder
< achow101> strange
< achow101> can you make the directory and try again?
< jonasschnelli> ok
< jonasschnelli> no change
< achow101> hmm
< achow101> I'll try on my mac tomorrow
< jonasschnelli> sure... I'll debug it a bit
< achow101> the relevant functiosn for that are _build_resources and _set_code_res_hash in signapple/sign.py
< jonasschnelli> _set_code_res_hash only hashes the file,.. (which is not there)
< wumpus> fanquake: that's really strange. the only explanation i can think of is that the PR was before hidden by github due to an admin action (e.g. some kind of account suspend that hides the persons PRs), and that was turned out, retrospectively revealing the PRs
< wumpus> the metadata archive (https://github.com/zw/bitcoin-gh-meta) doesn't have a file for it *at all* so it was not visible through the API either
< wumpus> MarcoFalke: https://github.com/bitcoin/bitcoin/pull/20720#issuecomment-748815722 beware, github-merge will still include your not-an-ACK as if an ACK :)
< bitcoin-git> [bitcoin] MarcoFalke pushed 5 commits to master: https://github.com/bitcoin/bitcoin/compare/e9efb64a0772...f061da288783
< bitcoin-git> bitcoin/master 93504da Hennadii Stepanov: ci: Fix COMMIT_RANGE variable value for PRs
< bitcoin-git> bitcoin/master 10af252 Hennadii Stepanov: ci: Drop Travis-specific way to set COMMIT_RANGE variable
< bitcoin-git> bitcoin/master c123892 Hennadii Stepanov: ci: Drop Travis-specific workaround for shellcheck
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #20697: ci: Fix COMMIT_RANGE variable value for PRs (master...201218-range) https://github.com/bitcoin/bitcoin/pull/20697
< wumpus> i wish there was a way to add 'blocked on issue/PR X' metadata to github issues
< wumpus> this could make it immediately visible what (for the project) the current blockers are and why something is blocked
< wumpus> some age-old issue tracking systems (maybe Trac?) had this
< bitcoin-git> [bitcoin] laanwj pushed 4 commits to master: https://github.com/bitcoin/bitcoin/compare/f061da288783...68c7acf6bb77
< bitcoin-git> bitcoin/master fadd402 MarcoFalke: psbt: Assert that tx has a value in UpdatePSBTOutput
< bitcoin-git> bitcoin/master fa7e803 MarcoFalke: Remove unused MakeOptional
< bitcoin-git> bitcoin/master fa4435e MarcoFalke: Replace boost::optional with std::optional
< bitcoin-git> [bitcoin] laanwj merged pull request #20671: Replace boost::optional with std::optional (master...2012-stdOpt) https://github.com/bitcoin/bitcoin/pull/20671
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #19426: refactor: Change * to & in MutableTransactionSignatureCreator (master...2007-refactorSign) https://github.com/bitcoin/bitcoin/pull/19426
< bitcoin-git> [bitcoin] hebasto opened pull request #20735: script: Remove outdated extract-osx-sdk.sh (master...201221-doc) https://github.com/bitcoin/bitcoin/pull/20735
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #20736: rpc: Replace boost::variant with std::variant for RPCArg.m_fallback (master...2012-rpcStdVariant) https://github.com/bitcoin/bitcoin/pull/20736
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/68c7acf6bb77...1077c93a3410
< bitcoin-git> bitcoin/master 11a3272 Michael Dietz: test: run mempool_resurrect.py even with wallet disabled
< bitcoin-git> bitcoin/master 1077c93 MarcoFalke: Merge #20692: test: run mempool_resurrect.py even with wallet disabled
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #20692: test: run mempool_resurrect.py even with wallet disabled (master...mempool-resurrect-to-miniwallet) https://github.com/bitcoin/bitcoin/pull/20692
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #20737: test: Add missing assignment in mempool_resurrect.py (master...2012-testAssign) https://github.com/bitcoin/bitcoin/pull/20737
< luke-jr> are there any Windows devs among us at this point? <.<
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #20738: [0.20] final rc2 backports (0.20...2012-20rc2) https://github.com/bitcoin/bitcoin/pull/20738
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #20739: [0.19] final rc2 backports (0.19...2012-19rc2) https://github.com/bitcoin/bitcoin/pull/20739
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #20737: test: Add missing assignment in mempool_resurrect.py (master...2012-testAssign) https://github.com/bitcoin/bitcoin/pull/20737
< bitcoin-git> [bitcoin] MarcoFalke reopened pull request #20737: test: Add missing assignment in mempool_resurrect.py (master...2012-testAssign) https://github.com/bitcoin/bitcoin/pull/20737
< wumpus> luke-jr: did we ever ? besides sipsorcery who is still active
< wumpus> looks like macos has the most developer enthousiasm these days, with linux close second
< wumpus> luke-jr: btw please rebase #14066 it'd be nice to get that in soon
< gribble> https://github.com/bitcoin/bitcoin/issues/14066 | gitian-linux: Build binaries for 64-bit POWER by luke-jr · Pull Request #14066 · bitcoin/bitcoin · GitHub
< sipsorcery> Nicolas Dorier is most likely still Windows, so I'm not completely alone.
< luke-jr> ☺
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/1d7243d30720...2556a973ed58
< bitcoin-git> bitcoin/master b23349b Ben Carman: rpc: Add missing description of vout in getrawtransaction help text
< bitcoin-git> bitcoin/master 2556a97 MarcoFalke: Merge #20731: rpc: Add missing description of vout in getrawtransaction he...
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #20731: rpc: Add missing description of vout in getrawtransaction help text (master...fix-getrawtransaction-help-text) https://github.com/bitcoin/bitcoin/pull/20731
< achow101> jonasschnelli: I found and fixed the bug in signapple. had an extra line I thought I removed. should work now
< phantomcircuit> so what should i do with `slow-unit` test cases? those are test cases that took longer than the timeout to fuzz (i think the timeout is huge btw, like 2 minutes or something)
< wumpus> phantomcircuit: either ignore them for fuzzing or figure out *why* they take so long, I do think there are some test cases that take unreasonably long
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/2556a973ed58...cc2a5ef9b2ff
< bitcoin-git> bitcoin/master fab46b3 MarcoFalke: test: Fix restart node race
< bitcoin-git> bitcoin/master cc2a5ef Wladimir J. van der Laan: Merge #20683: test: Fix restart node race
< bitcoin-git> [bitcoin] laanwj merged pull request #20683: test: Fix restart node race (master...2012-testNodeStart) https://github.com/bitcoin/bitcoin/pull/20683
< phantomcircuit> wumpus, now that i look more it's kinda hard to do that actually cause it just puts them in the top directory with no indication of which test it was running
< wumpus> phantomcircuit: wait, which tests are you running?
< wumpus> for the unit tests you can enable verbose debugging e.g. -l all (great overkill) or -l test_suite
< sipa> wumpus: it's the process_message fuzz test i believe
< wumpus> sipa: okay no idea about that one
< phantomcircuit> wumpus, im generating more fuzzing targets
< phantomcircuit> sipa, i think im going to have to create working directories for each test to capture the crash, timeout, slow tests cause they just get dumped into the cwd without any indication of which target they were running
< sipa> phantomcircuit: perhaps add that functionality to test/fuzz/test_runner.py
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/cc2a5ef9b2ff...f4ac48d30a84
< bitcoin-git> bitcoin/master 1fbb8b6 Hennadii Stepanov: script: Remove outdated extract-osx-sdk.sh
< bitcoin-git> bitcoin/master f4ac48d Wladimir J. van der Laan: Merge #20735: script: Remove outdated extract-osx-sdk.sh
< bitcoin-git> [bitcoin] laanwj merged pull request #20735: script: Remove outdated extract-osx-sdk.sh (master...201221-doc) https://github.com/bitcoin/bitcoin/pull/20735
< phantomcircuit> sipa, there's some other improvements i can make to the test_runner.py as well so yeah i guess that makes sense
< sipa> 100000 is not very many iterations for some of the fuzz targets
< phantomcircuit> sipa, there's a max_total_time parameter but it doesn't quite seem to work right
< phantomcircuit> or maybe im misunderstanding how it's supposed to work
< phantomcircuit> sipa, right it reads and executes the corpus once before checking the total runtime, but it includes the time to run the corpus in the total time
< phantomcircuit> potential for footgun there is high
< sipa> phantomcircuit: another useful feature would be setting a certain % of runs to get the -use_value_profile=1 flag
< phantomcircuit> sipa, what's that flag do?
< phantomcircuit> oh records the inputs to cmp
< phantomcircuit> interesting
< sipa> phantomcircuit: libfuzzer has two mechanisms to measure "coverage", one that doesn't really like at the values used in branching, and one that does
< sipa> doing that occasionally lets it more rapidly discover more varied inputs, but not necessarily inputs that lead to more actual coverage'
< phantomcircuit> sipa, yeah i was going to change the generation to use max_total_time, and then run each target with -jobs=$par -workers=$par instead of running each target with a single worker but all in parallel
< sipa> hmm, why?
< phantomcircuit> what happens now with high parallelism is some of the targets finish instantly and some take forever but only use 1 core
< sipa> it should start new ones, i think?
< phantomcircuit> sipa, there's ~150 tests, i have 128 cores, so what happens is i end up with like 16 of them running after about 30 seconds
< sipa> hmm, this works very differently than what i imagined i would do
< sipa> i'd just make it start N threads, pick a target randomly from the set you selected, and if one ends, start a new one
< sipa> that would be a lot better than what -jobs/-workers in libfuzzer can do, as it's not restricted to one single configuration for all threads
< sipa> when fuzzing manually i usually run with some distribution of -max_len and -use_value_profile parameters
< sipa> instead of all the same
< phantomcircuit> sipa, also if you run with -runs=10000 and -workers=128 some of the workers finish faster
< sipa> yeah
< phantomcircuit> i think the only way to actually be efficient is the max total time
< sipa> i just want something that runs forever, and starts random targets, and starts new ones when one ends
< phantomcircuit> sipa, yeah that's basically what i was going to do
< phantomcircuit> sipa, my concept with -workers/jobs was to run each of the things in sequence but race to finish
< wumpus> yes with fuzzing it's kind of hard to determine when to terminate anyway?
< phantomcircuit> the timing thing is annoying though cause it means i have to like benchmark loading the corpus or something
< wumpus> what would it be, 'we covered all code paths 100%'?
< phantomcircuit> wumpus, i think you're just never done is really the answer
< wumpus> right
< phantomcircuit> sipa, i guess what i can do is run without -jobs/workers and just keep starting new jobs randomly
< sipa> yeah
< phantomcircuit> for the targets with a large existing corpus they really have to run for quite a while to be useful
< CubicEarth> sipa: fyi - when my addnode is on the lan, core 0.19 pulled about 750 MB from the local node, and the other 750 MB spread out among the other wan peers, during a ~10 day sync catchup
< bitcoin-git> [bitcoin] achow101 closed pull request #20638: build: Fix macOS code signing by pre-allocating space for the code signature during gitian build (master...mac-codesign-fixed-alloc) https://github.com/bitcoin/bitcoin/pull/20638
< phantomcircuit> CubicEarth, that sounds about right just cause the lan peer is going to respond to requests with lower latecy
< CubicEarth> yeah, I was just curious what the ratio would be
< wumpus> it's interesting that it's almost 50/50
< wumpus> there's no logic at all to prefer local peers
< sdaftuar> CubicEarth: did you notice if your lan peer ever got disconnected during that catchup time?
< phantomcircuit> sipa, also good to know -workers with -merge just means it does the same thing a many many times for no reason
< phantomcircuit> so the existing test_runner.py does the right thing
< sipa> phantomcircuit: yeah, it just invokes itself a few times with a simple scheduler
< bitcoin-git> [bitcoin] practicalswift opened pull request #20740: fuzz: Update FuzzedDataProvider.h from upstream (LLVM) (master...bump-FuzzedDataProvider.h) https://github.com/bitcoin/bitcoin/pull/20740