< achow101>
from what I can tell, iOS binaries are macho as well
< achow101>
wumpus: are you currently writing a verification tool?
< wumpus>
searching for oid 1.2.840.113635.100.9.1 and 1.2.840.113635.100.9.2 (as appear in the CMS OIDs) gives some matches
< wumpus>
nah just trying to see if asn1crypto.cms in python can make sense of it, and it can, actually verifying anything is far away :):
< achow101>
I'll put together something to extract the hashes so we can at least verify those
< wumpus>
which oid has the hashlist/hashdict?
< wumpus>
or is that those 9.1/9.2
< achow101>
1.2.840.113635.100.9.2 probably
< wumpus>
they're seperate so it makes sense to be those two
< achow101>
9.1 is the string for a plist file
< achow101>
it does contain the hashes in there
< achow101>
9.2 is a sequence containing a single bye string that matches the code directory hash
< wumpus>
yes that makes sense if it's the same data in some legacy format and the new format
< achow101>
the hash in 9.1 is also trunctaed to 20 bytes
< wumpus>
probably fine to ignore the .1 one
< achow101>
yep
< sipa>
by tomorrow morning i'm sure you guys have a tough compatible signer
< sipa>
the day after i expect one that doesn't need a private key
< wumpus>
ok, after some more digging: 1.2.840.113635.100.9.1 is appleHashAgility/SEC_OID_APPLE_HASH_AGILITY/kCMSAttrAppleCodesigningHashAgility, .2 is appleHashAgilityV2/SEC_OID_APPLE_HASH_AGILITY_V2/kCMSAttrAppleCodesigningHashAgilityV2, .3 is appleExpirationTime/SEC_OID_APPLE_EXPIRATION_TIME/kCMSAttrAppleExpirationTime
< wumpus>
sipa: haha yesss
< * fanquake>
is just waiting for the TLDR
< achow101>
i'm sure that once a compatible signer is written, apple will change codesigning
< sipa>
fanquake: apple opensource codesign_allocate code behaves differently from the used binary; we found a workaround
< sipa>
fanquake: now achow101 and wumpus are trying to reverse engineer the signature format
< achow101>
so that we can implement an independent verifier, and maybe an independent signer
< fanquake>
Sounds like everything is under control
< fanquake>
Apple being a pain in the arse as per usual
< fanquake>
is the dmg signature being invalid only an issue when opening the .dmg on macOS Big Sur?
< achow101>
it's invalid everywhere
< achow101>
when the verifier tries to hash the binary, it's literally the wrong hash
< fanquake>
what's meant to happen? The .dmg opens fine here
< fanquake>
or does it just fail silently / log something
< sipa>
the signature is on the binary, not on the dmg i think?
< achow101>
try opening it
< achow101>
the app itself
< fanquake>
right, not the .dmg
< achow101>
if you open the dmg and do the drag into Applications, it should give you the warning
< fanquake>
I can run the app just fine from the cmd line
< sipa>
fanquake: both when doing the actual signing and when attaching the sig to the binary, some changes to the binary need to be made first; the apple tool does this *slightly* differently than their published source code, so the resulting binary after attaching isn't exactly identical to what the codesigning app produced
< achow101>
fanquake: do "codesign -v Bitcoin-Qt.app"
< achow101>
with rc3 installed
< fanquake>
Yea i see the invalid signature warnings.
< fanquake>
It seems like macOS will only stop you from opening the .app though. You can still run Bitcoin-Qt with an invalid signature
< fanquake>
Although that's not useful for us
< fanquake>
By Bitcoin-Qt I mean: Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt
< dhruvm>
Networking question: Are legal CIDR netmasks always 1s followed by 0s? i.e. mask=255.96.0.0 is an illegal mask?
< dhruvm>
I am trying to reason whether given two CIDR subnet representations, it is possible to have partial address overlap between them.
< sipa>
no, either they don't overlap at all, or one is a subset of the other
< sipa>
or identical to it
< dhruvm>
I see. So, the mask is always 1s followed by 0s then?
< dhruvm>
They seem to be represented as /32 /24 in most places, but some places I've seen 255.255.0.0 etc which has me confused
< sipa>
yeah, in CIDR the netmask has to be 1s and then 0s
< sipa>
you can have netmask that are not CIDR though
< sipa>
those aren't used in practocr afaik though
< sipa>
*practice
< dhruvm>
thanks sipa
< sipa>
yw
< jonasschnelli>
I upgraded my system and my gitian LXC console can no longer access the internet (it can fetch via apt-cacher-ng, but not the depends packages). Any idea?
< jonasschnelli>
sudo lxc-execute -n gitian -f var/lxc.config -- sudo -u root -i -- ping 8.8.8.8 <- times out
< sipa>
jonasschnelli: there is that snippet of code you need to run to set up network interfaces
< jonasschnelli>
sipa: what do you have in gitian-builder/var/lxc-config for the key lxc.net.0.ipv4.address?
< sipa>
i just followed the docs, on a fresh debian install
< jonasschnelli>
and my br0 has inet 10.0.2.2?! shouldn't it be 10.0.3.1?!
< wumpus>
on my gitian host the host br0 is 10.0.3.2
< wumpus>
though as long as they're used consistently throughout it doesn't really matter what the exact addresses are
< wumpus>
achow101: phantomcircuit: FWIW here's my (very makeshift) python script from last night that uses macholib and asn1crypto to extract and print the signature information from a macos binary: https://gist.github.com/laanwj/a0e00bcd3fe4cd2aa1c0803e91310495
< wumpus>
all dependencies are pure python as it is now; but for actual verification it's hard to get around openssl i'm afraid
< Talkless>
Anyone using QtCreator to work with bitcoin core? It's code model does not find any of bitcoin core includes, all underlined. I've created project as Autotools.
< wumpus>
i think you need to manually configure somehow what paths to look in for include files
< wumpus>
it's just the pyasn naming instead of asn1crypto, because of course, crypto can never be confusing enough
< jamesob>
hm, when importing a watch-only wallet, is there any risk to using `scantxoutset` to find the oldest UTXO, and then using that as the base block to pass to `rescanblockchain` for that wallet? you might miss the transactions for some spent coins, but wondering if that even matters.
< wumpus>
we'll get there, apparently it is a very similar mechanism as PE authenticode (windows code signing)
< wumpus>
jamesob: you'd say the resulting balance and utxo set should be the same, would definitely test though
< harding>
jamesob: you'd have to guess the range for scantxoutset, though. If you guess too low then it'll look like the wallet is empty.
< jamesob>
you mean the upper end of the range? I'd just assume go to tip; I'm just thinking you could use `scatxoutset` to truncate the lower bound
< harding>
jamesob: sorry, I meant the derived address range, which I think non-Bitcoin Core software usually calls the gap limit.
< jamesob>
harding: oh! gotcha; yeah I was talking about the rescan range in height units
< harding>
$ bitcoin-cli help scantxoutset | grep '"range"'
< harding>
"range": n or [n,n], (numeric or array, optional, default=1000) The range of HD chain indexes to explore (either end or [begin,end])
< wumpus>
maybe the best way forward is to add apple macho verification support to signify and upstream it; conceptually it's similar and there's bound to be so much shared code, e.g. this is the signature extraction and check for signed PE files: https://github.com/ralphje/signify/blob/master/signify/signed_pe.py
< wumpus>
okay, scrap that, of couuurse pyasn1 (in contrast to asn1crypto) can't parse the signature data, well let's see if i can get it to work with that
< michaelfolkson>
Will we want a BIP 44 coin type index for Signet?
< michaelfolkson>
Perhaps too early. Perhaps should only be considered once Signet is more stable/mature
< wumpus>
wow i got it to validate
< wumpus>
michaelfolkson: the consensus seems to be that new testnets will simply use testnet's parameters
< michaelfolkson>
wumpus: Are there any parameters (descriptors, HD wallet, network etc) where this could pose problems? I guess it only gets complicated if the user is switching between testnet and signet
< michaelfolkson>
And presumably not many will be doing that
< michaelfolkson>
(regularly switching back and forth I mean)
< luke-jr>
michaelfolkson: signet is for testing only. any problems, you get to pick up pieces ;)
< luke-jr>
user is assumed to know what he is doing
< michaelfolkson>
luke-jr: I guess... would still be nice to help them avoid unnecessary gotchas perhaps
< achow101>
wumpus: nice! does it rely on openssl for the validation?
< wumpus>
achow101: it uses oscrypto.asymmetric, no idea what that uses internally
< achow101>
can you update the gist?
< wumpus>
sure
< wumpus>
achow101: updated, note that it doesn't verify the intermediate certificate chain yet, only the final signatures on the SignerDatas
< achow101>
great
< wumpus>
I tried to use certvalidator to do that, but it gets into problems, even when manually providing the sorted certificate chain to the path validator it barks on some apple-specific extension *shrug*
< wumpus>
it's cool to find so much code that *almost* does what i want but is just that little bit off
< wumpus>
ok, updated gist again, it can now optionally validate the certificate chain (however, this required a one-line patch to certvalidator unfortunately)
< sipa>
awesome
< jonasschnelli>
shall I try to gbuild and detach-sign 20638?
< jonasschnelli>
or are you guys in the middle of fixing the internet?
< sipa>
#20638
< gribble>
https://github.com/bitcoin/bitcoin/issues/20638 | build: Fix macOS code signing by pre-allocating space for the code signature during gitian build by achow101 · Pull Request #20638 · bitcoin/bitcoin · GitHub
< sipa>
jonasschnelli: yeah, try please
< wumpus>
jonasschnelli: i guess we're working on a tool that checks the signature (i wrote the part that validates the CMS envelope, achow101 the part that checks hashes against the binary), this doesn't replace or delay #20638
< gribble>
https://github.com/bitcoin/bitcoin/issues/20638 | build: Fix macOS code signing by pre-allocating space for the code signature during gitian build by achow101 · Pull Request #20638 · bitcoin/bitcoin · GitHub
< achow101>
wumpus: are you sure that it's validating the right thing? the data being passed into verify is more than just the hash of the CodeDirectory
< achow101>
it also seems like it hashes the data too
< wumpus>
I'm pretty sure it validtes the right thing, the SignerInfos contain signed_attrs that has the list of hashes
< wumpus>
my script doesn't validate the actual hashes against the binary of course
< achow101>
right, that's the part I'm adding to it
< sipa>
do you know where to find the page sizes used?
< achow101>
i'd be surprised if the verify is able to pull out the hashes from the "signed_attrs" because that seems like some apple specific stuff
< achow101>
sipa: it's embedded in CodeDirectory
< achow101>
or do you mean in the source?
< sipa>
achow101: just wondering if you can account for that (so that it'd keep working if a different page size is used at some point)
< achow101>
yep, definitely accounting for that
< sipa>
wumpus, achow101: easy to test if it's validating the right thing; if you damage the signature, does it fail?
< sipa>
or better, damage the message being signed
< wumpus>
to get out your hashes data you need for attr in signerinfo['signed_attrs']: if attr['type'].native == SEC_OID_APPLE_HASH_AGILITY_V2: print(attr['values'][0].contents)
< wumpus>
that gets the byte data, which i guess is simply concatenated hashes
< wumpus>
it can extract everything, it's of no help parsing it of course
< achow101>
there's a field named "message_digest" that I'm trying to grab
< achow101>
that's not an apple specific thing apparently
< achow101>
apparently the hash is in the sig 3 times in 3 different fields
< wumpus>
attr['type'].native == 'message_digest'
< achow101>
also, oscrypto uses openssl in the backend
< jonasschnelli>
stupid question: why can't we just use the open source variant of codesign (if that exists)? to match codesign_allocate?
< achow101>
mmm, what if we cut off the allocated space?
< achow101>
jonasschnelli: there's no open source variant of codesign. codesign also requires codesign_allocate to be signed by apple, so we can't use the open source variant of that either
< sipa>
jonasschnelli: codesign only accepts apple-signed versions of codesign_allocate
< achow101>
"If the CODESIGN_ALLOCATE environment variable is set, it identifies a substitute codesign_allocate tool used to allocate space for code signatures in Mach-O binaries. This is used by Xcode SDK distributions to provide architectural support for non-native platforms such as iPhones.
< achow101>
The system will not accept such substitutes unless they are specially signed (by Apple)."
< jonasschnelli>
hmm.. apple style again
< sipa>
we can also try patching our codesign_allocate to roubd to 0x2000
< sipa>
it's a bit more brittle, but i don't see how it can fail
< jonasschnelli>
should we also file a bug at apples issue tracker for a long term fix of this?
< achow101>
they have one?
< achow101>
hmm, what if we put in an empty SuperBlob?
< achow101>
jonasschnelli: if you do "printf "\xfa\xde\x0c\xc0" | dd of=dist/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt bs=1 seek=26391600 count=4 conv=notrunc" and then try to sign, does it work?
< * jonasschnelli>
doing...
< jonasschnelli>
still getting: dist/Bitcoin-Qt.app: invalid or unsupported format for signature
< achow101>
darn
< jonasschnelli>
what if we overwrite the bytes with an "valid" signature we extract from a successful signing?
< bitcoin-git>
[bitcoin] sipa opened pull request #20644: Add patch to make codesign_allocate compatible with Apple's (master...202012_codesign_allocate_segalign) https://github.com/bitcoin/bitcoin/pull/20644
< sipa>
jonasschnelli: can you try that? ^
< sipa>
(entirely untested)
< achow101>
jonasschnelli: would probably work
< jonasschnelli>
okay.. I'll clear the cache and run it
< jonasschnelli>
achow101: I mean you could probably replace the zero signature (the allocation) with the largest signature you find in 0.12
< jonasschnelli>
maybe the tool only validates the header or so,.. so combining with the allocation hack might work?
< jonasschnelli>
trying sipa patch now
< achow101>
with the -f option, codesign will only warn if the existing sig is invalid. but I guess it needs to be properly formatted first
< achow101>
maybe not even warn
< achow101>
so you could try to apply the rc3 sig to the pr's build and then sign that
< jonasschnelli>
achow101: what does codesign_allocate do with the existing sig? Just expand?
< jonasschnelli>
(I mean in your PR)
< achow101>
yes
< jonasschnelli>
that could work...
< achow101>
wumpus: huh, I guess it does sign the whole "signed_attrs" thing and not just the message digest as I previously thought
< jonasschnelli>
But how would I apply the sig?
< achow101>
run detached-sig-apply.sh locally
< jonasschnelli>
yes... I could try that
< sipa>
my PR doesn't work
< sipa>
fixing
< jonasschnelli>
achow101: can I take a unsigned.tar.gz from master,... manually apply the sig (detached-sig-apply.sh) on linux, execute your patch ${CODESIGN_ALLOCATE} -i ${BINARY_RESULT} -a x86_64 ${SIG_SIZE_EST} -o ${BINARY_RESULT} and then try to sign on OSX?
< jonasschnelli>
without gitian at all?
< jonasschnelli>
I can take the same cc_tools (form the cache)
< achow101>
to test, the sig creation, yes
< jonasschnelli>
sipa: so 6e9f0fa7f7ce95cc4ddf0e5cc4c49ab75361efe7 is it?
< sipa>
jonasschnelli: yup
< sipa>
unfortunately, this working isn't a guarantee it will always work
< sipa>
because the build may randomly be 0x2000 aligned
< jonasschnelli>
sipa: could we detect this easly?
< achow101>
jonasschnelli: is there some kind of dev testing code signing cert? I'd like to be able to test this stuff
< jonasschnelli>
achow101: sure. I can try to make you one
< achow101>
There's no apple default?
< jonasschnelli>
I don't know
< achow101>
ah, i'll look into it later
< jonasschnelli>
I guess if I hand you out a different certificate, end uses would have no plan/idea to differenciate those
< achow101>
yeah
< sipa>
you won't have a certificate chain leading back to apple
< sipa>
or perhaps one which doesn't convey codesigning rights
< achow101>
I wouldn't be surprised if apple didn't let you sign with such a cert
< jonasschnelli>
sipa: so your cctools patch is only relevant for the signing step? I don't need to rebuild the actual unsigned release build?
< jonasschnelli>
it only affects detached-sig-apply.sh, right?
< achow101>
yes
< sipa>
jonasschnelli: right
< achow101>
ah, there's instructions for a self signed cert. this may work
< jonasschnelli>
but! gitian-osx-signer.yml does not build the depends? So I need to clear the cache and do the release build anyways
< sipa>
ah right
< sipa>
you need it because the signing step uses the binaries produced in the unsigned build step
< jonasschnelli>
yes
< sipa>
ugh
< sipa>
my patch fails
< sipa>
sorry i will test this more and get back
< jonasschnelli>
sipa: sure. No hurry
< achow101>
at this point it feels like the long term solution is to write our own codesigning tool
< sipa>
sounds like you're well on the way to do that :)
< sipa>
if you have a tool that can verify signatures, signing doesn't sound too far away
< achow101>
signing is probably 2 orders of magnitude harder than verification
< achow101>
the current unknown things being how to generate Info.plist, CodeResources file, and the internal requirements blob
< jonasschnelli>
achow101: fatal error: /home/jonasschnelli/bitcoin-depends2/depends/x86_64-apple-darwin20/native/bin/x86_64-apple-darwin20-codesign_allocate: size for '-a x86_64 258611375' not a multiple of 16
< sipa>
round up to a multiple of 16
< jonasschnelli>
stat -c %s signed-app/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt
< jonasschnelli>
-> 26698160
< achow101>
oh, I totally forgot about that didn't I
< jonasschnelli>
shall I try to use this detached signature to build the release?
< achow101>
to be clear, you did codesign-allocate with the overestimate, applied some other signature, then signed
< achow101>
yes, try that
< jonasschnelli>
yes...
< jonasschnelli>
I did: /home/jonasschnelli/bitcoin-depends2/depends/x86_64-apple-darwin20/native/bin/x86_64-apple-darwin20-codesign_allocate -i dist/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt -a x86_64 258611376 -o dist/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt
< jonasschnelli>
tar -czf bitcoin-0.21.0rc3-osx-unsigned.tar.gz ./
< jonasschnelli>
./detached-sig-apply.sh bitcoin-0.21.0rc3-osx-unsigned.tar.gz ../bitcoin-detached-sigs/osx (<--- the git repo for the signatures)
< sipa>
ok it seems copy-pasting the "git diff" output introduced some space/tab changes...
< achow101>
jonasschnelli: awesome
< jonasschnelli>
let me 2bl check first
< jonasschnelli>
Yes. I think I have done that correctly
< jonasschnelli>
I mean I can just try to apply this signature I created on my mac no on linux with ./detached-sig-apply.sh
< jonasschnelli>
right?
< jonasschnelli>
no need for gitian
< sipa>
i believe so
< achow101>
yea
< sipa>
jonasschnelli: ok gitian manages to compile native_cctools with my patch now
< jonasschnelli>
The dbl-signing approach seems not to work
< sipa>
if you want to try (if the other approach works, also fine of course)
< achow101>
jonasschnelli: how so?
< jonasschnelli>
What I did:
< jonasschnelli>
1. I replace the signature in the detached-signtaure local repo with the one I just successfully did on my mac (after codesign_allocate on linux
< jonasschnelli>
2. on linux, I executed ./detached-sig-apply.sh bitcoin-0.21.0rc3-osx-unsigned-2.tar.gz ../bitcoin-detached-sigs/osx
< jonasschnelli>
(bitcoin-0.21.0rc3-osx-unsigned-2.tar.gz is from the rc3 release, unsigned, untouched)
< jonasschnelli>
(../bitcoin-detached-sigs/osx hold the new signature)
< jonasschnelli>
then downloaded signed-app/Bitcoin-Qt.app to my mac
< achow101>
jonasschnelli: for the final apply, you need to use the one that had the codesign-allocate and bad sig thing
< jonasschnelli>
achow101: ah...
< jonasschnelli>
okay.. let me try that
< jonasschnelli>
achow101: still getting /Users/jonasschnelli/Desktop/Bitcoin-Qt.app: invalid Info.plist (plist or signature have been modified)
< jonasschnelli>
Maybe I did something wrong...
< jonasschnelli>
let me upload some files so you can test it
< jonasschnelli>
I can't use the rc3 sig for the cctool patch test,.. since it's master now and has different version numbers.
< jonasschnelli>
So trying to sign independent
< jonasschnelli>
I hoped by deleting the cctools only in the cache, I can avoid rebuilding the whole dependencies.... but it looks like it builds everything again
< jonasschnelli>
I have to continue tomorrow (10pm here,... will start again in 11h)...
< bitcoin-git>
[bitcoin] jonatack opened pull request #20646: p2p: do not disconnect post-verack sendaddrv2 on signet (master...signet-keep-post-verack-sendaddrv2-peers) https://github.com/bitcoin/bitcoin/pull/20646
< achow101>
sipa: does your pr gitian build? I have a self signed cert so I can test codesigning now
< sipa>
achow101: yes
< sipa>
i did a gitian build of the unsigned stage myself