< fanquake>
sipa: was looking at PRs tagged as backports, and that was one. 2nd PR like that recently that has just "appeared "
< sipa>
phantomcircuit: so, create a new directory $NEWDIR, fill it with just the qa-assets' seeds for the process_message fuzzer, and then run FUZZ=process_message ./src/test/fuzz/fuzz -merge=1 -use_value_profile=1 $NEWDIR $DIRWITHYOURNEWSEEDS
< luke-jr>
>_<
< fanquake>
I opened an issue with GitHub ~5 days ago asking why it was happening. No reply from them yet.
< phantomcircuit>
sipa, hmm so by default this is just putting the new test cases into the same directory, copying the qa-assets seeds is to prevent them from being deemed duplicates right (i saw your comment on the github issue about merging)
< phantomcircuit>
i'd probably need to have a separate copy of them for that to really work i guess (although through git magic i guess i do)
< sipa>
phantomcircuit: when fuzzing, did you start with the existing qa-assets, and fuzzed on top, or did you start from scratch?
< phantomcircuit>
sipa, i started with the existing qa-assets
< sipa>
ok
< phantomcircuit>
sipa, having clearly not read enough about this, is the the sanest way to use cpu time, or should i use new seeds and merge them into the corpus?
< phantomcircuit>
or should i copy the corpus, fuzz on the corpus and then merge back
< sipa>
i'd do the last
< phantomcircuit>
sipa, btw test_runner.py runs each test just once by default
< phantomcircuit>
which i guess makes sense for actually running them
< phantomcircuit>
but for fuzzing isn't super useful i think
< phantomcircuit>
oh i see --generate
< phantomcircuit>
sipa, ok this is working nicely i think
< phantomcircuit>
sipa, ok it takes a very long time to do the merge lol
< sipa>
phantomcircuit: it'll run every seed from both inputs once to determine their coverage
< phantomcircuit>
sipa, i also have a bunch of 'slow-unit-' files
< phantomcircuit>
sipa, interestingly some of the tests run through 100,000 runs really fast and others do not, i guess that makes sense though
< bitcoin-git>
[bitcoin] hebasto opened pull request #20734: build: Make platform-specific targets available for proper platform builds only (master...201221-targets) https://github.com/bitcoin/bitcoin/pull/20734
< jonasschnelli>
achow101: for the codesign test,... should I set the env var CODESIGN_ALLOCATE to the binary we have in the tar.gz (the oss one)?
< achow101>
jonasschnelli: yeah
< sipa>
the same codesign_allocate as will be used to attach the sig
< sipa>
achow101: would it make sense for your tool to also do the attaching?
< achow101>
sipa: yes, I plan on adding detaching and attaching options to it
< achow101>
just need to figure out how to format the detached sig
< sipa>
cool!
< sipa>
i expect that will be easy compared to everything else you've done
< achow101>
yes, it's mostly just a design decision
< achow101>
and whether we want to maintain compatibility with the current process
< sipa>
achow101: the actual codesign tool also has a detached format
< achow101>
sipa: it can't be attached to a binary
< achow101>
although maybe with some modifications it could
< sipa>
ah
< achow101>
the main problem is the separate CodeResources file. but codesign's detach option embeds that in the signature. I'll have to see if it's possible to do that in the embedded signature so that we don't have to deal with multiple files
< sipa>
jonasschnelli: if you're going to try with rc3, use the codesign tool from the gitian build (without my recent patch)
< jonasschnelli>
achow101: how do I pass the identity to your tool?
< achow101>
ah, so it can't fetch the cert from MacOS's keychain. you need to export it as a .p12 and give the path, no -s
< achow101>
there's instructions in docs/certificates.md
< jonasschnelli>
-h is out of date. :)
< * jonasschnelli>
reading...
< achow101>
updating documentation is not a strong suit
< jonasschnelli>
achow101: after entering the .p12 password (your tool) I'm getting FileNotFoundError: [Errno 2] No such file or directory: '/Users/jonasschnelli/Desktop/bitcoin-0.21.0rc3-osx-unsigned/dist/Bitcoin-Qt.app/Contents/_CodeSignature/CodeResources'
< jonasschnelli>
My execution is: ${CODESIGN} sign /Users/jonasschnelli/Desktop/mycert.p12 -f --file-list ${TEMPLIST} "$@" "${BUNDLE}"
< jonasschnelli>
(I'll remove the -s and removed the help/parameter-check in detached-sig-create)
< achow101>
does that file exist after the tool fails?
< jonasschnelli>
achow101: nope... there is no _CodeSignature folder
< achow101>
strange
< achow101>
can you make the directory and try again?
< jonasschnelli>
ok
< jonasschnelli>
no change
< achow101>
hmm
< achow101>
I'll try on my mac tomorrow
< jonasschnelli>
sure... I'll debug it a bit
< achow101>
the relevant functiosn for that are _build_resources and _set_code_res_hash in signapple/sign.py
< jonasschnelli>
_set_code_res_hash only hashes the file,.. (which is not there)
< wumpus>
fanquake: that's really strange. the only explanation i can think of is that the PR was before hidden by github due to an admin action (e.g. some kind of account suspend that hides the persons PRs), and that was turned out, retrospectively revealing the PRs
< bitcoin-git>
bitcoin/master b23349b Ben Carman: rpc: Add missing description of vout in getrawtransaction help text
< bitcoin-git>
bitcoin/master 2556a97 MarcoFalke: Merge #20731: rpc: Add missing description of vout in getrawtransaction he...
< bitcoin-git>
[bitcoin] MarcoFalke merged pull request #20731: rpc: Add missing description of vout in getrawtransaction help text (master...fix-getrawtransaction-help-text) https://github.com/bitcoin/bitcoin/pull/20731
< achow101>
jonasschnelli: I found and fixed the bug in signapple. had an extra line I thought I removed. should work now
< phantomcircuit>
so what should i do with `slow-unit` test cases? those are test cases that took longer than the timeout to fuzz (i think the timeout is huge btw, like 2 minutes or something)
< wumpus>
phantomcircuit: either ignore them for fuzzing or figure out *why* they take so long, I do think there are some test cases that take unreasonably long
< phantomcircuit>
wumpus, now that i look more it's kinda hard to do that actually cause it just puts them in the top directory with no indication of which test it was running
< wumpus>
phantomcircuit: wait, which tests are you running?
< wumpus>
for the unit tests you can enable verbose debugging e.g. -l all (great overkill) or -l test_suite
< sipa>
wumpus: it's the process_message fuzz test i believe
< wumpus>
sipa: okay no idea about that one
< phantomcircuit>
wumpus, im generating more fuzzing targets
< phantomcircuit>
sipa, i think im going to have to create working directories for each test to capture the crash, timeout, slow tests cause they just get dumped into the cwd without any indication of which target they were running
< sipa>
phantomcircuit: perhaps add that functionality to test/fuzz/test_runner.py
< phantomcircuit>
sipa, there's some other improvements i can make to the test_runner.py as well so yeah i guess that makes sense
< sipa>
100000 is not very many iterations for some of the fuzz targets
< phantomcircuit>
sipa, there's a max_total_time parameter but it doesn't quite seem to work right
< phantomcircuit>
or maybe im misunderstanding how it's supposed to work
< phantomcircuit>
sipa, right it reads and executes the corpus once before checking the total runtime, but it includes the time to run the corpus in the total time
< phantomcircuit>
potential for footgun there is high
< sipa>
phantomcircuit: another useful feature would be setting a certain % of runs to get the -use_value_profile=1 flag
< phantomcircuit>
sipa, what's that flag do?
< phantomcircuit>
oh records the inputs to cmp
< phantomcircuit>
interesting
< sipa>
phantomcircuit: libfuzzer has two mechanisms to measure "coverage", one that doesn't really like at the values used in branching, and one that does
< sipa>
doing that occasionally lets it more rapidly discover more varied inputs, but not necessarily inputs that lead to more actual coverage'
< phantomcircuit>
sipa, yeah i was going to change the generation to use max_total_time, and then run each target with -jobs=$par -workers=$par instead of running each target with a single worker but all in parallel
< sipa>
hmm, why?
< phantomcircuit>
what happens now with high parallelism is some of the targets finish instantly and some take forever but only use 1 core
< sipa>
it should start new ones, i think?
< phantomcircuit>
sipa, there's ~150 tests, i have 128 cores, so what happens is i end up with like 16 of them running after about 30 seconds
< sipa>
hmm, this works very differently than what i imagined i would do
< sipa>
i'd just make it start N threads, pick a target randomly from the set you selected, and if one ends, start a new one
< sipa>
that would be a lot better than what -jobs/-workers in libfuzzer can do, as it's not restricted to one single configuration for all threads
< sipa>
when fuzzing manually i usually run with some distribution of -max_len and -use_value_profile parameters
< sipa>
instead of all the same
< phantomcircuit>
sipa, also if you run with -runs=10000 and -workers=128 some of the workers finish faster
< sipa>
yeah
< phantomcircuit>
i think the only way to actually be efficient is the max total time
< sipa>
i just want something that runs forever, and starts random targets, and starts new ones when one ends
< phantomcircuit>
sipa, yeah that's basically what i was going to do
< phantomcircuit>
sipa, my concept with -workers/jobs was to run each of the things in sequence but race to finish
< wumpus>
yes with fuzzing it's kind of hard to determine when to terminate anyway?
< phantomcircuit>
the timing thing is annoying though cause it means i have to like benchmark loading the corpus or something
< wumpus>
what would it be, 'we covered all code paths 100%'?
< phantomcircuit>
wumpus, i think you're just never done is really the answer
< wumpus>
right
< phantomcircuit>
sipa, i guess what i can do is run without -jobs/workers and just keep starting new jobs randomly
< sipa>
yeah
< phantomcircuit>
for the targets with a large existing corpus they really have to run for quite a while to be useful
< CubicEarth>
sipa: fyi - when my addnode is on the lan, core 0.19 pulled about 750 MB from the local node, and the other 750 MB spread out among the other wan peers, during a ~10 day sync catchup
< bitcoin-git>
[bitcoin] achow101 closed pull request #20638: build: Fix macOS code signing by pre-allocating space for the code signature during gitian build (master...mac-codesign-fixed-alloc) https://github.com/bitcoin/bitcoin/pull/20638
< phantomcircuit>
CubicEarth, that sounds about right just cause the lan peer is going to respond to requests with lower latecy
< CubicEarth>
yeah, I was just curious what the ratio would be
< wumpus>
it's interesting that it's almost 50/50
< wumpus>
there's no logic at all to prefer local peers
< sdaftuar>
CubicEarth: did you notice if your lan peer ever got disconnected during that catchup time?