< gmaxwell>
oh hmph. made some pgoress but stuck again. :(
< gmaxwell>
now this looks like local corruption from running out of space. 2016-03-24 00:00:19 ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=-1, nPos=0)
< sipax>
ugh!
< sipax>
that means the block data wasn't written but the index entry for it was
< sipax>
remind me to investigate the write order during flush in such a case
< * sipax>
goes into standby mode
< gmaxwell>
well it may have been out of space and the write could have failed. Our out of space handling is not so great always.
< sipax>
it should not go write a database entry for a file it failed to create
< sipax>
is it a pruned node?
< gmaxwell>
no.
< sipax>
in that case, certainly not
< gmaxwell>
what the heck, even after logging that a bunch of times it's making progress again.
< gmaxwell>
okay, actually the readblockfromdisk is being triggered by getblock rpcs being called by p2pool
< gmaxwell>
so this might just be an artifact of running getblock on a block we have headers for but no data.
< sipax>
gmaxwell: no, it shouldn't
< sipax>
getblock RPC checks whether the BLOCK_HAVE_DATA flag is present for that block index entry
< gmaxwell>
well my log is full of
< gmaxwell>
2016-03-23 23:58:18 Received a POST request for / from 127.0.0.1:51001
< gmaxwell>
and now that it's caught up, no more..
< morcos>
gmaxwell: sipax: looks like thats just the error that happens, the RPC only checks the BLOCK_HAVE_DATA flag if you're pruning. Ha!
< jonasschnelli>
encryption: would it be stupid to use the identity keypair (used for Auth / MITM protection) in a ECDH scheme to encrypt a dh key-exchange for further encryption?
< jonasschnelli>
I think we should not use the "static" identity key for encryption.
< jonasschnelli>
It would expose the shared ecdh secred (AES256key) to known-plaintext-attacks.
< jonasschnelli>
But not sure if using the identity key for a quick ECDH keyexchange (known-plaintext-attacks are pretty impossible) would make sense.
< gmaxwell>
It would be stupid beyond all comprehension.
< gmaxwell>
The encryption should be ephemerally keyed. There is no need for the key to outlive the session.
< gmaxwell>
What I suggested earlier-- which is a pretty standard and well studied architecture is to use randomly generated keys to establish a secure channel (with authenticated encryption like AES-GCM or chacha20-poly1305); then inside if-- if some identity is in use-- you use that identity to authenticate the secure session you just established. E.g. by signing the hash of the shared session key.
< sipax>
gmaxwell: that is MitMable?
< gmaxwell>
... No.
< sipax>
oh, no!
< jonasschnelli>
gmaxwell: how would you pass the random generated symmetric cipher key to "the other side"? ECDH channel with the identity key?
< gmaxwell>
Please please stop trying to use ecdh with long lived keys. This is almost always a severe design mistake.
< gmaxwell>
Each side sends a new, randomly generated public key to the other side. The session keys (there will be more than one, likely four-- auth and encrypt in each direction) are the output of ECDH with those keys, fed through a KDF.
< jonasschnelli>
gmaxwell: Okay. How would you protect from MITM. By auth with the identity key AFTER the encryption channel is setup?
< gmaxwell>
if identification is in use, inside the encrypted channel you send a signature of the session id (a hash of the ephemerial key from ecdh which also commits to all the channel parameters) using the relevant identity.
< sipax>
what do you mean by channel parameters?
< jonasschnelli>
gmaxwell: how would you protect from attacking(DOSing) the encryption request (key generation!) if the auth is after the enc?
< jonasschnelli>
Also fingerprinting
< jonasschnelli>
(the later is probably fine if the 4 key are generated randomly)
< gmaxwell>
key generation is very fast, faster then verifying your digital signatures you propose for authentication.
< gmaxwell>
Fingerprinting what?
< * jonasschnelli>
always though key generation is much more CPU intense then ecdsa verify.
< sipax>
jonasschnelli: if you always use freshly generated ecdh keys, there is no fongerprinting
< jonasschnelli>
gmaxwell: fingerprinting the node.
< jonasschnelli>
Well,.. right. We could ban a node after a failed encryption request.
< sipax>
i don't think it can fail
< jonasschnelli>
sipax: I mean the identity check afterwards
< sipax>
jonasschnelli: it's much cheaper for them to go ask us a bunch of random blocks, with far higher costs
< sipax>
if they want to dos
< gmaxwell>
jonasschnelli: no, it's fundimentally easier to generate a new keypair than to verify a ecdh signature. They're pretty close in timings, because we use a constant time implementation for the former.
< jonasschnelli>
But this would basically mean, every peer can request encrypted sessions... (which i'm not sure if its good or bad).
< gmaxwell>
they should, and they should all be encrypted so as not to distingush the ones using authentication.
< gmaxwell>
and this can be made faster than the existing transport (or at least not considerably worse).
< sipax>
jonasschnelli: for ECDSA, verify > sign >>> keygen
< jonasschnelli>
sipax: I see. Thanks!
< sipax>
jonasschnelli: computing the pubkey for a generated private key is similar to signing
< gmaxwell>
well if by keygen you mean generate a pubkey too, then sign == keygen, pretty much.
< gmaxwell>
and this is faster than verify.
< sipax>
same order of magnitude in any case
< jonasschnelli>
gmaxwell: So. The peer would pass one pubkey for the dh secret calculation and the (static) identity pubkey (static) for auth to the other peer. Right?
< jonasschnelli>
(doesn't the current[and most] ec keygen algos do a dummy sign/verify to ensure validity?)
< sipax>
jonasschnelli: if by auth you mean the signing of the session key, indeed
< sipax>
jonasschnelli: bitcoin does
< gmaxwell>
the only software ever created that does that is bitcoin core.
< gmaxwell>
as far as I can tell.
< gmaxwell>
and it's only sensible to do that for payment addresses, it wouldn't be done here.
< gmaxwell>
first an encrypted, authenticated channel would be brought up using a ephemeral pubkey sent in each direction. Then _inside_ that channel, the participants could identify themselves by signing the session-id.
< jonasschnelli>
only signing the session key would not allow the remote peer to identify which identity is signing. Wouldn't it require at least the hash of the identity pubkey?
< gmaxwell>
Techincally the signature and message its signing is enough, you can just get the pubkey from that. Though it could send the pubkey its using.
< jonasschnelli>
Ah. Right pubkey recover.
< gmaxwell>
but there are better protocols for identifying than just sending a signature-- ones that do not leak the identity of at least one participant.
< jonasschnelli>
0-knowlage?
< jonasschnelli>
What about allowing auth without enc? We probably presume encryption when authenticating.
< gmaxwell>
jonasschnelli: I think there is no utility to identifying without encryption.
< jonasschnelli>
Agree. The current auth proposal would also require signing (auth) each message (which is resource hungry).
< gmaxwell>
thats unacceptably slow.
< jonasschnelli>
encryption before authentication allows to keep a auth-session.
< sipax>
jonasschnelli: note that authentication is used in two meanings here; there is identification (done by signing the session key) and MAC (which you still need, but ks covered by using an authenticated mode of encryption such as AES-GCM)
< gmaxwell>
(and signining indivigual messages provides non-repudiation, which is usually not what you want; except in specific instances where you do want it.)
< jonasschnelli>
I see.
< sipa>
jonasschnelli: so there is "auth" on each message, but it's done with a symmetric shared session key, rather than the identity key
< jonasschnelli>
And I guess it would be acceptable if the remote peer can fingerprint the requesting peer because it reveals its identity after the encryption channel is established.
< gmaxwell>
jonasschnelli: it's better than the other way around at least. preferably no one could fingerprint anyone; but it's not clear how possible that is.
< jonasschnelli>
sipa: Right. But we just need to make sure the symmetric shared session key was shared between the right parties and not Mitmdled.
< sipa>
jonasschnelli: i think you probably want a per-IP identity, or perhaps just randomly generated identities by default, and allow a peer to ask "hey, you're identity X, right? prove it"
< gmaxwell>
Other than the "connect to my own server" case it's not clear to me exactly how the identification would be used.
< jonasschnelli>
sipa: But that would not work for DHCP nodes (SPV)?
< sipa>
jonasschnelli: so perhaps a separate private non-random configurable host key, which is only used if requested by the peer
< sipa>
that means no TOFU, though
< gmaxwell>
we don't want to create cases where nodes can be tracked easily, we've worked to remove identifying information from the protocol... having a protocol where there is mutual auth, say between your own 'server' and 'client' or between your own hosts, having the requester leak something the allows them to be correlated, but only inside an encrypted channel which they think is to their trusted pe
< gmaxwell>
er, would be minimally harmful.
< gmaxwell>
sipa: I don't know what use tofu would have... and what happens when a node goes away (say, gets deleted) and a new one is at that address? now you need an expiration mechenism.
< jonasschnelli>
gmaxwell: at least you could setup a honeypot-like peer that waits until other peers connect, request encryption and identify themself. You could than combine this data with other fingerprint possibilities (mempool and stuff like that).
< gmaxwell>
jonasschnelli: the party initating authentication would be the only one to lose privacy, so a honey pot would not work; unless it also successfully mitmed hosts that clients were attempting to connect to.
< jonasschnelli>
But somehow I think either you have a MITM attack possibility or you reveal ones identity. Maybe some zero knowledge prove method could be used here (not familiar with these)
< gmaxwell>
There are probably multiple usecases which would identify the channel in different ways; but the same secure channel mechenism would work.
< gmaxwell>
jonasschnelli: I linked a likely sutable identification protocol above.
< * jonasschnelli>
is reading the JFK proposal. Thanks to gmaxwell
< gmaxwell>
I am not suggesting we specifically use that, as much as pointing out that it's possible to do better... but even a very simple protocol does what we want, I think.
< sipa>
jonasschnelli: you could also avoid the problem, and by default not have identification, and only have a "prove to me you're X before continuing" message inside the encryptee channel
< sipa>
perhaps even using symmetric crypto, so it only works with a preshared symmetric key
< jonasschnelli>
sipa: Yes. Agreed. Auth would only be necessary or useful if the both peers know each other (preshared key, same owner, SPV<->node stuff).
< gmaxwell>
identification, when you say auth it easily gets confused with the authentication part of encryption which is necessary and cannot be omitted when encryption is used. :)
< jonasschnelli>
A preshared preshared symmetric key would also work. Right. IMO its just easier to share two pubkeys instead of one "symmetric key".
< jonasschnelli>
gmaxwell: Can you explain the differences between the identity authentication and the encryption authentication? Why would the later be required if both parties stay anonyme to each other?
< jonasschnelli>
Once you exchanged encryption ephemeral pubkeys, MITM should no longer be possible?
< gmaxwell>
jonasschnelli: any use of encryption _must_ come with symmetric-key authentication, or otherwise the encryption will be mallible and an attacker can corrupt the channel in ways that leak information about the content.
< gmaxwell>
e.g. they flip a bit, and instead of instantly hanging up, the application is fed some gibberish, and then based on exactly how it responseds/doesn't respond the attacker learns something about the state of the connection and the prior encrypted data.
< gmaxwell>
so the encryption must comes with a message-authentication-code keyed at the same time as the encryption, so any corruption is detected and the connection terminated without revealing anything, ideally before even decrypting the data.
< jonasschnelli>
gmaxwell: sharing the encryption ephemeral pubkeys, calculate symmetric secret and hash the complete communication (starting with both pubkey in deterministic order), compare the hashes after decrypt..
< gmaxwell>
jonasschnelli: that isn't sufficient.
< sipa>
jonasschnelli: encryption provides confidentiality, not integrity
< jonasschnelli>
hmm....
< * jonasschnelli>
is reading...
< Luke-Jr>
jonasschnelli: it might or might not make sense to coordinate with the new payment protocol encryption stuff
< Luke-Jr>
in BIP 75
< jonasschnelli>
Luke-Jr: thanks. Will check. Agree on the p2p 16x num range.
< sipa>
jonasschnelli, gmaxwell: the paper my const time aes is based on actually discusses aes-gcm (and easily has 4x the performance of the cbc case, as it can do 4 parallel encryption operations)
< jonasschnelli>
gmaxwell sipa: hmm... (having a hard time to understand this correctly): so you would require to generate two keypairs at the same time, one for the encryption sym. cipher key, one for the MAC key. Then you sould sha256_hmac the complete message traffic on both sides and compare to validate the integrity?
< sipa>
jonasschnelli: no, you use aes-gcm
< sipa>
which does encryption and mac simultanelusly
< jonasschnelli>
sipa: okay. I see. But the overall concept would be correct (just to understand the basics).
< jonasschnelli>
s/./?
< sipa>
though, conceptually, yes, you can use aes for encryption, and then do hmac-sha256 on the encrypted data for integrity
< gmaxwell>
There should be two "keys" if using authenticated encryption (like AES-GCM, chacha20-poly, or things coming out of CAESAR) one key in each direction; or four keys if constructing autenticated encryption from a cipher and hmac.
< jonasschnelli>
What I have problems to understand is how attacks would be possible if you would take the encryption ephemeral publey calculated sym. key for the MAC...
< sipa>
jonasschnelli: let's say you use aes-ctr
< sipa>
you know ctr?
< * jonasschnelli>
reading... got the essence. Yes.
< jonasschnelli>
streaming.
< sipa>
someone may not be able to decrypt without having the key, but if they can guess what the contents is of a particular message, they can modify it by xoring it with both what they think the decrypted data is, and xoring it with what they want the decrypted data to be
< sipa>
there are more complex bad things you can do with other encryption schemes
< jonasschnelli>
sipa: Got it! Thanks!
< sipa>
but ctr is a nice example, because it perfectly covers confidentially, but provides almost nothing else
< sipa>
so it clearly shows the distinction
< * jonasschnelli>
is reading the details of AES-GCM
< jonasschnelli>
sipa: But IIRC, your AES PR is CBC. So GCM would require another implementation while our codebase already supports SHA2 HMAC?
< sipa>
jonasschnelli: GCM is easy :)
< jonasschnelli>
sipa: for you or for all other lazy programmers.. :)
< gmaxwell>
sha2 is slow.
< jonasschnelli>
And I guess we could drop the message headers 4byte sha256 checksum once encrypted and MACed
< gmaxwell>
(AES-GCM is also not astonishingly fast without hardware AES, but should be faster than sha2)
< gmaxwell>
jonasschnelli: yes, and the result, with sutiable primitive selection would be _faster_ than the existing protocol, even though it was encrypted.
< jonasschnelli>
hah. Thats an argument.
< gmaxwell>
(faster in terms of cpu usage, it would likely have some more bandwidth overhead, but not a ton)
< jonasschnelli>
But I'm mostly thinking "in implementations". So,... I could write a first implementation that drops the 4byte header sha and uses SHA2 HMAC for the MAC.. just to have an implementation basepoint.
< jonasschnelli>
(in addition to the whole encryption keysharing stuff)
< jonasschnelli>
I could be the guy that works on the code frontend while I require analysis and conceptual support from gmaxwell/sipa and others. This could lead to a pratical p2p encryption IMO.
< sipa>
gmaxwell: my current const aes needs 270 cycles/byte; making it do 4 in parallel is trivial and would make it need 72; researchers claim they can do ~10 using simd
< sipa>
and ~3 with aes-ni
< gmaxwell>
sipa: I think with a.. right.
< gmaxwell>
See also the motivationes behind the CAESAR competition.
< gmaxwell>
sipa: lol one of the CAESAR entries is based on keccak and is named "keyak". That totally isn't going to cause any confusion...
< sipa>
gmaxwell: i was just reading the keyak v2 paper
< btcdrak>
I've asked a cryptographer to look at #7689
< jonasschnelli>
Could we ask apoelstra?
< jonasschnelli>
I'm not sure if a standalone AES library (move sipas PR to a custom library) would make sense.
< jonasschnelli>
Adding more non-consensus stuff there will very likely lead to a openssl clone?!
< jonasschnelli>
And adding SHA512 but not SHA256?
< wumpus>
jonasschnelli: correct
< wumpus>
I don't think petertodd's concern is so much about the library, but more about 'how much review doees this get'
< jonasschnelli>
wumpus: after thinking about it. maybe a simple C bases AES library could have a reason to exists.
< sipa>
gmaxwell: what is the fastest authenticated encryption scheme implementation that you know? (assume no hardware support, but constant time)?
< wumpus>
jonasschnelli: sure, it could, but this is specifically a constant-time AES for wallet encryption
< jonasschnelli>
wumpus: Right. If the p2p encryption will be implemented once, it could be extended there (AES CGM).
< gmaxwell>
sipa: people who want that are using chacha20-poly1305. AES-GCM with hardware support is apparently somewhat faster (and much less power hungry); but without it the chacha20-poly1305 stuff is faster.
< wumpus>
it's e.g. not the fastest way to implement AES, because we specifically don't need that. There may not be many projects that want to tuse it at all, and there's a long tradition of copy/pasting crypto code anyhow....
< gmaxwell>
people who care about fast AES use hardware or use these parallel implementations which cannot be used for CBC mode, which compatiblity with the wallet encryption needs.
< wumpus>
gmaxwell: yes, exactly
< wumpus>
for the AES code I really have only one concern: is it correct
< wumpus>
for the random code I'm more scared because of the platform dependence and other uglyness involved
< sipa>
wumpus: agree
< gmaxwell>
currently we don't care at all about fast AES. We should somewhat care that it doesn't have stupid sidechannels. Later, if we care about fast AES for p2p encryption, it would be AES-GCM and end up with a CBC incompatible implementation in any case.
< wumpus>
yes, for P2P we could use a different implementation of AES, or not AES at all (chacha20 or so :p)
< gmaxwell>
right or whatever CAESER selects, depending on what that happens.
< wumpus>
yes
< sipa>
gmaxwell: that's 2 years out
< gmaxwell>
(most of the CAESER candidates are AES based in any case, mostly addressing security complaints about GCM, and expecting hardware support to address the obnoxiousness of constant time AES)
< * wumpus>
really likes chacha20, it's so small, efficient, and seemingly simple
< gmaxwell>
it has a mixed following, it's apparently enough faster on ARM and without AES-NI that many in the web world have considered it urgent to support... but apparently a lot of the datacenter folks are less than enthused since it ends up much more power hungry than hardware AES.
< wumpus>
yes, right, it's specifically efficient in sw, if there is a hw implementation of AES it'll win out
< jonasschnelli>
What MAC AAD would make sense for the p2p encryption? hash of the pubkey&pubkey? IP/Port?
< wumpus>
let's just wait for hw chacha20 *ducks*
< gmaxwell>
wumpus: well, its not like they have anything better to do with the transistors...
< wumpus>
gmaxwell: agree, the dark silicon problem
< wumpus>
then again - for bitcoin nodes we need to worry about consumer hw, not so much big datacenters
< wumpus>
I think it's safe to say there won't be so many bitcoin nodes in datacenters to rival the number of https implementations, let alone be a power usage concern
< gmaxwell>
well a lot of 'consumer hardware' has hardware aes now too, just not phones so much. :)
< gmaxwell>
but indeed.
< wumpus>
power usage on the other hand is mostly a phone concern. But yeah.
< gmaxwell>
in any case both, given good implementations, should be faster the current 'checksum'
< sipa>
jonasschnelli: the additional data is sent in every message
< sipa>
jonasschnelli: you would include the message length there, for example
< jonasschnelli>
sipa: hmm.. so hash would be too expansive?
< sipa>
hash of what?
< jonasschnelli>
hash of the encryption pubkeys or could also be a hash of the plaintext message.
< sipa>
what are you trying to do?
< sipa>
the additional data is a feature provided by authenticated encryption schemes; you don't need to use it if you have nk need for it
< jonasschnelli>
sipa: So do I got this right? You could use something like the current protocol version "70012" as AAD. And during decryption, you feed the TAG (processed AAD) together with the AAD into the decryption method and get a true or false?
< jonasschnelli>
And thanks for you time to explain that to me. :)
< gmaxwell>
you could-- generally it's used for additional per message data that must be sent cleartext for protocol reasons, but which you'd still like included in the authentication.
< jonasschnelli>
gmaxwell: Okay. Right. Message length would be ideal then.
< gmaxwell>
that would be the kind of thing; even that may not be necessary, if the length was wrong, the auth would fail in any case-- wrong amount of data in it.
< sipa>
jonasschnelli: i guess you can include the network magic to the aad
< jonasschnelli>
sipa: okay. Would that be much different then the encrypted message length?
< sipa>
jonasschnelli: i mean both... but there is no strong reason
< jonasschnelli>
okay.
< sipa>
essentially it guarantees that the length/magic came from someone with the session key
< jonasschnelli>
ok
< sipa>
aes-gcm uses 96-bit IVs, but they only require being unique for a given key
< sipa>
and the key is establishes through ecdh
< sipa>
so they can just be the message number, and don't need to be transmitted
< GitHub121>
bitcoin/master 146746b Alice Wonder: All files related to my RPM spec file project in one commit
< GitHub121>
bitcoin/master 0e4b50a Alice Wonder: Description of RPM directory
< GitHub121>
bitcoin/master 3ba07bd Wladimir J. van der Laan: Merge #7609: All files related to my RPM spec file project in one commit...
< GitHub36>
[bitcoin] laanwj closed pull request #7609: All files related to my RPM spec file project in one commit (master...master) https://github.com/bitcoin/bitcoin/pull/7609
< GitHub55>
bitcoin/master d6cc6a1 João Barbosa: Use CCoinControl selection in CWallet::FundTransaction
< GitHub55>
bitcoin/master b88e0b0 Wladimir J. van der Laan: Merge #7506: Use CCoinControl selection in CWallet::FundTransaction...
< GitHub17>
[bitcoin] laanwj closed pull request #7506: Use CCoinControl selection in CWallet::FundTransaction (master...enhancement/use-coin-control-selection) https://github.com/bitcoin/bitcoin/pull/7506
< sipa>
jonasschnelli: if you want to use a single negotiation to intialize encryption for both directions, you need to make sure that the IVs in bith directions are different, or you break the assumption that an attacker cannot see two different messages with the same key and iv
< sipa>
jonasschnelli: my suggestion here would be to have an encinit ("if you want, you can start sending encrypted messages to me and this is my key") and an encack ("yeah, every message after this one will be encrypted, and this is my key"), which you both send in both directions
< sipa>
jonasschnelli: that way, both directions have independent ECDH negotiations, and you also have no problems with synchronization
< sipa>
jonasschnelli: an alternative is thinking hard about synchronization, and making sure that the initiator and accepter of the encryption connection use separate IVs
< sipa>
jonasschnelli: reusing the key as IV is not very useful, i think
< sipa>
jonasschnelli: you probably want to include some text about how this interacts with version/verack (does encinit have to be the first message entirely? can it be done at any time?
< sipa>
jonasschnelli: your encrypted message protocol spec could make the authentication tag explicit (it's 16 bytes for aes-gcm, though we could decide to truncate it)
< jonasschnelli>
sipa: thanks for the review! Will "process" your points and come back to you with feedback.
< sipa>
openssh's chacha20-poly1305 aead at a high level
< sipa>
they encrypt the packet sizes too, but with a separate key
< sipa>
and then authenticate using the second key which is used for authentication and encryption of the data
< sipa>
which means that an attacker can't even see the message sizes, except by traffic analysis
< btcdrak>
Clever
< gmaxwell>
a little sad to require 4 byte lenghts.
< gmaxwell>
sipa: for identified links, we could do a nice thing with rekeying to make them strong against ECC breaks.
< sipa>
cfields: if we'd want to use something like that, i guess we want an api that allows a client to tell the comnection manager "let me know when you have X bytes"
< sipa>
cfields: which is oerhaps a cleaner abstraction than hardcoding a particular stream protocol?
< sipa>
jonasschnelli: another suggestion: i think we should drop the network magic... resynchronization after garbage hasn't been supported in a long time, it wastes (a tiny bit of) bandwidth, and makes the encrypted stream look decidedly identifiable (though if encrypted connections bootstrap as unencrypted ones, that isn't solvable)
< gmaxwell>
Alice connects to Bob, Alice sends, Nonce_a, and alice ephemeral pubky (AEP), bob sends nonce_b and BEP. ECDH runs, then a KDF that takes the two nonces, and ecdh output, and spits out a session id and two keys (one for each direction. Then Alice proposes, "I think you have identity X=H(bob pubkey, session_id)", if this is true, bob responds with "I am rekeying with IDX" and takes his cur
< gmaxwell>
rent encryption key and changes it to H(bob pubkey|enckey). When alice sees that message she updates her keys for the stream. Then also does the same herselve (sends a message, updates key). Then bob sends a signature with his pubkey. Then this authentication procedure can just be repeated every so many bytes in order to rotate the stream cipher keys.
< gmaxwell>
If the 'public keys' are kept private, the link has at least symmetric keyed integrity and confidentality, even with an ECC break.
< wumpus>
<sipa> which means that an attacker can't even see the message sizes, except by traffic analysis <- I think that would be quite essential for bitcoin, as so much can already be identified from the packet sizes
< wumpus>
certain blocks, certain transactions, etc
< gmaxwell>
I had assumed that the encrypted transport would encapsulate one or more length coded messages inside it... so mostly you'd just see the 'send units'. Doing this would alow avoiding that extra layering, however... and thus decrease overhead.
< gmaxwell>
though perhaps we should make the encoding support padding, so that messages can be padded up to particular size multiplies, to reduce the traffic analysis sidechannel.
< CodeShark>
is there a meeting today?
< gmaxwell>
yes, in about 20 minutes.
< gmaxwell>
You've suffered DST. :)
< btcdrak>
we dont suffer it over this side of the Atlantic until Sunday
< btcdrak>
gmaxwell: is it maybe worth adding a version byte to the encryption scheme to allow upgrades in the future?
< jonasschnelli>
gmaxwell: +1 for padding to reduce sidechannel identification.
< jonasschnelli>
sipa: do you mean dropping the network magic for all post-encryption-init messages?
< sipa>
also +1 to allowing multiple messages in a packet
< jonasschnelli>
You mean one header with a message count int followed by n payloads?
< sipa>
jonasschnelli: yes, it serves no purpose other than detecting accidental connection to the wrong network
< sipa>
jonasschnelli: or just a concatenation of payloads even
< jonasschnelli>
Okay. size + payload + size + payload, etc. That makes sense.
< sipa>
jonasschnelli: inside the aead
< gmaxwell>
that cuts down on the authentication overhead.
< jonasschnelli>
sipa: But I still try to understand your AES GCM 96bit IV solution. How would you create distinctable IV without transmitting in in the message?
< sipa>
jonasschnelli: use 0000 + counter for one side, use 1111 + counter for the other?
< jonasschnelli>
sipa: Are predictable IVs not a problem?
< gmaxwell>
but its maximum could just be the same as the largest message.
< sipa>
right
< jonasschnelli>
So. Changes i'll work on a) ecdh exchange for both direction, two sym. keys, fix synchro. b) use chacha20 poly1305 instead of AES GCM c) optimize message structure (drop magic, allow multiple payloads in the AEAD part)
< sipa>
jonasschnelli: no need to fix synchronization if both directions are independent
< jonasschnelli>
yes. I mean fix syncho by using two keypair, one per direction.
< gmaxwell>
great.
< jonasschnelli>
Has chacha20-poly1305 been proven to be stable? Reasonable amount of cryptanalysis? Or do we threat stuff out of DJB's kitchen as stable anyways? :)
< gmaxwell>
jonasschnelli: it's part of TLS and SSH.
< gmaxwell>
I think it's sutiable for our purposes.
< wumpus>
it's geting a lot of testing and review in ssh at least
< jonasschnelli>
Okay. And how complex is an implementation? Or would this lead to a revival of openssl in core?
< gmaxwell>
it's very simple.
< wumpus>
their implementation is very short, that's one thing that is so cool about it
< jonasschnelli>
what speaks again unsing openssl (p2p layer enc is non consensus)?
< jonasschnelli>
*against
< gmaxwell>
we're not taking 400,000+ lines of unreviewable, frequently CVEed code for this; I'd rather not have the functionality.
< wumpus>
attack surface, making it impossible to get rid of openssl dependency
< wumpus>
and you can already do it using stunnel
< gmaxwell>
I hope that this works so that even if we don't care about the confidentiality improvement, it's just simply faster than the existing transport.
< jonasschnelli>
Right. I think dropping the sha256 per message could be an improvement in performance.
< jonasschnelli>
combining multiple messages into one encrypted message also
< jonasschnelli>
The question is, how to "bundle" messages (example: bundle invs). When releasing them, etc. I mean for the implementation.
< sipa>
jonasschnelli: SendMessages is a function in main that computes a block of messages to send at once
< jonasschnelli>
Ah. But they get sent individually (one inv per msg)?
< sipa>
jonasschnelli: maybe you should look into how our current protocol works :)
< jonasschnelli>
haha. True.
< sipa>
no, an inv messages can contain up to 5000 invs
< jonasschnelli>
Okay. So where would we benefit from bundling messages in one encrypted message?
< wumpus>
it makes sense to make it possible, it benefits privacy
< sipa>
well, if you take as baseline an implementation that encrypts message sizes, there is no privacy benefit to bundling
< wumpus>
even if your first implementation doesn't actually use it, imo the protocol should support it
< gmaxwell>
phantomcircuit had a patch to add corking and uncoarking in the message handling loop, I presume something similar would be done for message batching.
< jonasschnelli>
agree
< wumpus>
ok, yes, no privacy benefit to bundling makes it less important to me
< sipa>
as they would be sent back-to-back, so traffic analysis would not let you discover independent messages
< gmaxwell>
e.g. we'd cork both the crypto and tcp, do all the message handling for a peer, then uncork both.
< sipa>
however, bundling does give a small bandwidth improvement, as you'd share a authentication tag acorss muktiple messages
< wumpus>
well, let's not micro-optimize before we have anything
< jonasschnelli>
tag is 12bytes. Right.
< gmaxwell>
not necessarily all that small, the auth is relatively large to our average message size.
< wumpus>
let's first make sure it is secure, then wonder about bandwidth optimization
< jonasschnelli>
Right. I'll try to update the Bips with what we have discusses and ask for another review.
< jonasschnelli>
*discussed
< wumpus>
trying to tackle everything at once is going to make this go nowhere, I'm afraid
< gmaxwell>
the bundling also makes padding easy, you just define a new message type "pad" that is just discarded on the other end. :)
< sipa>
jo16 if you want 128 bit security
< gmaxwell>
yea, iteration is fine.
< jonasschnelli>
should i say now: "perfect is the enemy of good". Yes? No. *duck*
< MarcoFalke>
meeting time?
< gmaxwell>
Meeting time.
< wumpus>
jonasschnelli: more like: keep your goal focused. Your goal is security and privacy, first. Make no dumb mistakes there ;)
< jonasschnelli>
RIght!
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu Mar 24 19:00:25 2016 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< wumpus>
jonasschnelli: scope creep is a very bitcoin-y thing
< wumpus>
ok, topic proposals?
< MarcoFalke>
python 3 and rpc tests?
< wumpus>
from last meeitng we have ACTION: merge #7575 (wumpus, 19:04:04) ... that was done
< btcdrak>
softfork #7648 status
< wumpus>
next was ACTION: review #7648 after #7575 is merged (wumpus, 19:09:39)
< Luke-Jr>
topic proposal: v0.11.3 and v0.10.5
< wumpus>
how is that going?
< wumpus>
#topic softfork #7648 status
< wumpus>
MarcoFalke: will get to python3 at the end, I think the softfork stuff is most important right now
< btcdrak>
#7648 has received some tested ACKs.
< MarcoFalke>
sure
< wumpus>
okay, good
< wumpus>
I think it needs more review still
< btcdrak>
time to crack the whip.
< wumpus>
for a softfork the bar is somewhat higher than for random pull #1234
< wumpus>
yes
< jonasschnelli>
7648 looks good and will also review it in details soon (during weekend)
< wumpus>
I mean I've seen some acks but very little comment on the actual code, that could be a sign everything is perfect, or people need to look better
< wumpus>
thanks jonasschnelli
< btcdrak>
MarcFalke: can you look as well please?
< wumpus>
that's way too much for a softfork release at least
< Luke-Jr>
what isn't clear to me is what the plan is for v0.11.3 being released - I assume wumpus is doing that at some point? - and whether I'm taking over for v0.10.5, or if wumpus wanted to do one final 0.10 before passing it off
< wumpus>
for a normal minor release it'd make more sense
< wumpus>
yes I'll do the 0.11.x release
< MarcoFalke>
7047 also includes a lot of "comment-fixed". I am not sure if they are useful in our branch
< gmaxwell>
whats deployment of 0.11.2 vs 0.12 look like right now?
< MarcoFalke>
I'd rather only have bugfixes
< wumpus>
I'll leave 0.10.x to you
< Luke-Jr>
are we doing a softfork release in the next month?
< sipa>
Luke-Jr: i say yes
< Luke-Jr>
ok, so then we should let these things wait until after that
< wumpus>
I think we should do a softfork release asap
< Luke-Jr>
or maybe I should go through it and make sure there's nothing important
< wumpus>
let's just get those stupid things reviewed and merged
< jonasschnelli>
As soon as #7648 has been reviewed more and merged
< wumpus>
jonasschnelli: and backports reviewed and merged ofc
< wumpus>
but yes 7648 is the first step
< jonasschnelli>
RIght
< Luke-Jr>
(also, depending on ease of backporting the softforks, I may decide to just give up on 0.10 support; tbd)
< btcdrak>
/Satoshi:0.12.0/1713
< btcdrak>
/Satoshi:0.11.2/1387
< jonasschnelli>
0.10 can be given up IMO
< * jonasschnelli>
is checking his seeder data
< gmaxwell>
Luke-Jr: I think absent someone requesting otherwise (and perhaps funding your effort) you probably should.
< Luke-Jr>
jonasschnelli: it's currently Gentoo stable, so at the very least I will need to bump that
< btcdrak>
The problem with 0.10 is we all need to put review time into the backports and that's difficult when we have trouble even reviewing backports for 0.11
< petertodd>
btcdrak: +1
< Luke-Jr>
btcdrak: most backports are trivial, but for softforks I agree if they're not a clean backport
< wumpus>
right
< btcdrak>
I think if there is a 0.10.5 release it should be extremely minimal... and frankly, backporting BIP68 wont be fun or easy.
< sipa>
bip68/112/113 are quite a bit of code
< wumpus>
let's spend our energy on moving forward
< jonasschnelli>
0.10 should be last priority. If someone offers to do that: fine.
< gmaxwell>
plus, a far backport which is done by one person and used by few more likely won't be materially more stable than a more recent release.
< Luke-Jr>
ok, so for 0.10 I'll see the complexity in the softfork backports, and if it's anything more than absolutely trivial, I'll just decide to drop it
< btcdrak>
Luke-Jr: even BIP68 backport to 0.11 was tricky, not a clean cherry-pick.
< Luke-Jr>
btcdrak: well, I'd be backporting to 0.10 *from* 0.11
< sipa>
we can still do critical security updates for 0.10 without choosing to backport sotfork
< wumpus>
sure
< jonasschnelli>
6.3% 0.10.x nodes
< btcdrak>
so on this topic, people need to also look at backport #7543
< Luke-Jr>
and for 0.11, sounds like the plan is to do a softfork-only release before any non-essential bugfix backports
< CodeShark>
+1 jonasschnelli
< petertodd>
sipa: and people who need a v0.10.x node for software compatibility reasons can generally run it behind a newer node
< sipa>
petertodd: also true
< wumpus>
#action people need to also look at backport #7543: [0.12] Backport BIP9, BIP68 and BIP112 with softfork
< sipa>
petertodd: specifically, behind a 0.11 or 0.12 pruned one :)
< petertodd>
sipa: yes! that too
< Luke-Jr>
#action 0.10: attempt to backport softforks, and if non-trivial, discontinue support
< Luke-Jr>
#action 0.11: no non-essential fix backports until softfork release
< Luke-Jr>
#action bump Gentoo stable to 0.11 (from 0.10)
< sipa>
Luke-Jr: support is already discontinued according to policy, only critical biffixes
< sipa>
*bugfixes
< CodeShark>
I'd say discontinue support for 0.10 unless someone wishes to provide resources for it
< wumpus>
well if luke-jr wants to try to backport the softfork that's up to him
< Luke-Jr>
sipa: policy only dictates guaranteed support, not guaranteed to end
< sipa>
wumpus: sure, but i don't think that's very relevant here
< wumpus>
no need to even discuss it here :p
< wumpus>
right
< btcdrak>
#action 0.11 review #7716 Backport BIP9 and softfork for BIP's 68,112,113
< Luke-Jr>
any more topics?
< sipa>
as soon as we have 0.12.1, i will rebase segwit on top, so segnet can benefit from its bip9/68/112/113 support
< cfields>
can give a quick update on the net refactor
< wumpus>
I'd say spend the rest of the meeting time reviewing #7648 :p
< wumpus>
sure cfields!
< wumpus>
#topic cfields net refactor
< sipa>
please, stick to the net refactor, not the gross one
< cfields>
i finally have a working full-featured branch. it's a mess code-wise, but i think it should be approachable in the next day or two after some code movement. I expect to be looking for concept review next week
< gmaxwell>
cfields: start taking, I can't take the wait.
< wumpus>
lol
< wumpus>
cfields: I've been testing it on a node btw for last week, seems to be stable and working
< wumpus>
haven't looked at details of performance etc but no crashes or insane log entries
< btcdrak>
sipa: you can cherry-pick from #7543 in that case
< cfields>
wumpus: that's great to hear. lots has changed since then, but the core is the same
< cfields>
i think i have an idea of how to handle code review, but i'd like to get some concept acks first...
< wumpus>
you got my concept ack at the boston meeting
< sipa>
and mine
< cfields>
there's a huge mix of code movement, a new lib, abstraction, etc. I think i have a plan for getting it slotted in
< wumpus>
great
< cfields>
ok. i can start PRing chunks into core as i go, then
< Luke-Jr>
cfields: practical to add block streaming? ;)
< cfields>
first step is getting the net stuff instanced. that's a lot of movement without much functional difference
< gmaxwell>
do not creep cfield's scope. :P
< wumpus>
please, let's replace the current things first, then add additional features
< jonasschnelli>
hah
< wumpus>
example gmaxwell
< wumpus>
exactly*
< cfields>
i think that can start going in parallel
< cfields>
gmaxwell: heh, i've adjusted scope so many times now that i'm certainly not budging again :p
< jonasschnelli>
The net refactor will be very hard to review already. Lets keep it as simple as possible.
< wumpus>
that's good, stand your ground , people here are awful :)
< cfields>
turns out the absolute minimum scope for net refactor is already enormous
< jonasschnelli>
Indeed.
< wumpus>
yes it'll already be challenging to get this done before 0.13 I think, but we have to try
< Luke-Jr>
gmaxwell: indeed, just hope we don't need to refactor it again after
< sipa>
Luke-Jr: we absolutely will
< cfields>
wumpus: yes, it's much later than I was hoping for. If it slips past 0.13, I'll understand.
< sipa>
but at different layers :)
< wumpus>
right, this is only the bottom layer
< cfields>
sipa: well the hope is that it's been split up into chunks that can be changed in the future much easier
< sipa>
cfields: yes, that's what abstractions are for
< jonasschnelli>
topic: Should we shorty touch how we should proceed with sipas CT AES implementation? Extract to library? Yes/No?
< cfields>
there are a few hacks that i'm not going to mess with for now. For ex, even though it's all instanced, there's still a g_connman that's used liberally as opposed to the clean way of passing the instance around
< wumpus>
meh
< cfields>
i'll start working on a list of todos so that some of the decisions are more clear
< petertodd>
jonasschnelli: extract is a great idea I think - don't have to be fancy, just enough to continue to set a better standard than "roll our own"
< sipa>
cfields: just PR it! i want to see the code :)
< cfields>
sipa: i'm still frantically coding it :)
< sipa>
ah.
< cfields>
jonasschnelli: nooooooo. everyone shield your eyes from that :)
< wumpus>
the code is actually buildable and works
< sipa>
now i have a moral obligation to actually go look, right?
< wumpus>
well test at least :)
< btcdrak>
cfields: love that last commit message :)
< cfields>
sipa: with the understanding that it's basically my /tmp :)
< jonasschnelli>
cfields: We all know how code during implementation can look. No shame for it!
< wumpus>
yes :)
< jonasschnelli>
petertodd: I'm also for extracting.
< jonasschnelli>
Simple autotools setup, add as subtree place on bitcoin/libctaes (or similar)
< wumpus>
#topic CT AES library
< Luke-Jr>
what is "CT" for here?
< wumpus>
I don't think extracting it into a library has to affect our use of it
< petertodd>
jonasschnelli yup, subtree is fine
< wumpus>
Constant Time
< jonasschnelli>
constant time
< Luke-Jr>
ah
< sipa>
i feel like it's more a code snippet than a library
< gmaxwell>
constant time.
< wumpus>
yes, it's more like a snippet
< gmaxwell>
"Petertodd raised verification concerns for the internal AES implementation. I could suggest some strategies on that." is what I wrote earlier. :P
< wumpus>
crypto has a rich tradition of copy/paste
< cfields>
fwiw, I plan on doing the same with libbtcnet
< jonasschnelli>
wumpus: I think placing stuff in libraries follows a modular approach which is desirable for Bitcoin IMO
< Luke-Jr>
+1 for libctaes
< cfields>
i have a tiny/crappy makefile that i use to build it externally, but I plan to just copy/paste it into Core and keep it in sync
< petertodd>
sipa: well, to be clear, you did write that code snippet from scratch yourself right?
< wumpus>
you could stil include it in a library, but please don't compilicate the build
< Luke-Jr>
cfields: libbcnet I hope; what does the BTC unit size have to do with networking? :P
< gmaxwell>
in any case, I'm not convinced that the standard test vectors prove the code correct. Beyond soliciting outside review, which btcdrak has done. I was going to suggest trying some mutation testing to see if some additional better test vectors can be constructed.
< jonasschnelli>
sipa: Its a snipped now, ... but we all know how snippets evolve over time.
< cfields>
Luke-Jr: when i googled "libp2p" some etherium thing came up :)
< sipa>
petertodd: yes, though the logic gates for the sbox come from a paper
< wumpus>
I think we should separate concerns here: getting the code into a library, and getting the damn code reviewed
< wumpus>
the first is easy, just work
< Luke-Jr>
cfields: oh great, let's just integrate that! :P
< petertodd>
sipa: right, so it's independent work, which means it's crypto we wrote, which means no matter how small it is, we should at least make a token effort to try to let independent use and review :)
< Luke-Jr>
wumpus: I don't feel competent to review crypto code :<
< wumpus>
Luke-Jr: me neither.
< wumpus>
Luke-Jr: so we'll have to find someone that can, btcdrak had some ideas.
< gmaxwell>
You're all competent to review if the code is constant time or not.
< sipa>
or that it has test coverage
< jonasschnelli>
What about apoelstra?
< gmaxwell>
And probably no one is competent to review if it's correct, fortunately AES is a permutation, which does make confidence building via testing easier.
< wumpus>
but not if it's correct AES, or has some weird edge cases, mathatical inconsistencies, or other wacky side channels
< wumpus>
right
< sipa>
and notice it has no tables or branches, so it would be exceedingly hard to make it broken in a very small number of cases
< jonasschnelli>
IMO that all speak for splitting it off into a own library.
< petertodd>
gmaxwell: the problem here, is I don't know enough about the problem to know if I'm actually competent to review that
< gmaxwell>
(by no one, I mean no human. :) )
< btcdrak>
petertodd: I asked isis to take a look on a paid basis.
< petertodd>
btcdrak: good!
< gmaxwell>
petertodd: the requirement is that there should be no leakage of the secret state (key or data) in the timing of the program on any hardware we support.
< btcdrak>
that's @isislovecruft from Tor for anyone that doesnt know.
< * Luke-Jr>
volunteers to do the splitting-off-into-library work if sipa doesn't want to
< sipa>
anyway, i'll turn it into a .c/.h; if someone else wants to actually do the work for autotools etc, go ahead.
< jonasschnelli>
Luke-Jr: I already have started that. :)
< wumpus>
Luke-Jr: this will again trigger the subtree-or-external discussion, I can't take that another time
< wumpus>
Luke-Jr: and linux distributions messing it up etc
< gmaxwell>
it will not be external.
< Luke-Jr>
wumpus: seems that discussion should be project-global, not per-lib
< jonasschnelli>
sipa: I have checked to code and it looks pretty Cish.
< Luke-Jr>
subtree-with-configure-option seems to work
< sipa>
imho, it can be a separate repository without being a library
< sipa>
just copy it into your project if you neeed it
< wumpus>
right.
< sipa>
it's a single file and has no intention to grow beyond that
< gmaxwell>
nor any utility to grow beyond that.
< jonasschnelli>
meh. Adding a build setup should be trivial and would hurt nobody?
< cfields>
sipa: is there enough non-standard stuff to necessitate a buildsystem? or would a simple makefile do?
< wumpus>
the API will never change, the implementation will hopefully also never change (excluding critical problems)
< sipa>
cfields: it will be a single C89 file with no dependencies beyond stdint
< Luke-Jr>
stdint isn't C89, though?
< gmaxwell>
wumpus: the only 'change' I could imagine for it is wrapping it with something that uses AESNI when available instead.
< sipa>
Luke-Jr: it indeed isn't, which is why i mention it :)
< cfields>
sipa: ah great
< gmaxwell>
well C89+stdint, or C99...
< gmaxwell>
stdint is almost universal even in terrible embedded complilers.
< jonasschnelli>
Are there no plans to extend the "lib" with more stuff? Like for a possible p2p encryption?
< wumpus>
and if not it's easy enough to define the types yourself
< Luke-Jr>
jonasschnelli: well, then the name libctaes would be wrong
< sipa>
jonasschnelli: i'd do that on another layer
< jonasschnelli>
Right.
< wumpus>
no, jonasschnelli, that should be something else yet again
< sipa>
jonasschnelli: it would still rely on aes primitives
< gmaxwell>
This pressure to constantly create "libraries" to dump things in is streesful for me. A well constructed library is a serious amount of work in and of itself.
< wumpus>
ctaes would be constant time aes :-) I don't think you want/need that for the p2p
< petertodd>
gmaxwell: I may have to use a climbing analogy to get my point accross here :) sure, I can easily read a book on knots and belay systems and feel like I _should_ know everything I need to know to do that stuff safely, but the consequences of failure are high enough - and the field unusual enough - that I'm going to insist I check things out with others generally accepted as experts first - who cares how irrational ...
< jonasschnelli>
Okay. Then we might agree in a subtree with just a .c/.h instead of a autotools setup.
< petertodd>
... that feels. I think general standards for low-level crypto implementations tend in that direction.
< wumpus>
jonasschnelli: yes
< petertodd>
jonasschnelli: yes, just a .c/.h is fine! like I said, bare minimum to set a precedent that we're trying to invite outside review
< Luke-Jr>
subtree with just .c/.h until some future time when someone decides to lib-ify it seems ok
< wumpus>
jonasschnelli: that's very common for crypto
< sipa>
and maybe for performance reasons, it makes sense to use a more complicated faster implementation, and maybe also warrants more maintainance afterwards if it's going to be adopted by other prijects for the purpose of p2p encryption
< jonasschnelli>
okay.
< gmaxwell>
petertodd: considering that basically all AES-CBC software (not AESNI) out there is gratitiously timing vulnerable, and we were going to take such a patch ourselves before; I feel you're compent enough to review and increase confidence, if not alone to achieve perfection.
< Luke-Jr>
sipa: seems likely other wallets may use it
< jonasschnelli>
agree on bitcoin/libctaes?
< wumpus>
and just copy/subtree the thing into your project and include it into your build system, all the dependency micro management isnt' worth it for everything
< sipa>
Luke-Jr: they absolutely may!
< jonasschnelli>
Luke-Jr: right. I have two projects where i will use it immediately.
< wumpus>
jonasschnelli: bitcoin-core/libctaes it would be then
< gmaxwell>
as I noted before, this implementation is not super fast. We have no need to make it super fast. It may be of idependant interest to others because it is quite small.
< sipa>
Luke-Jr: but for wallets, it has no performance requirements
< jonasschnelli>
wumpus: ah. Right. The moving.
< sipa>
Luke-Jr: which is why it's easy, and needs no dependencies on assemblers or instruction sets, or simd extensions, or ...
< * Luke-Jr>
not sure libctaes makes sense as bitcoin-core as opposed to just bitcoin, but meh
< petertodd>
gmaxwell: yeah, I probably am - I also learned how to do climbing/caving ropework by buying a book on it, some equipment, and going out to find some trees to practice on. Doesn't mean it was a good idea. :) just having a separate repo with a bare .c/.h goes a long way to making our intentions clear and having good precedents, even if pragmatically we need a solution right now.
< wumpus>
jonasschnelli: which reminds me, I think I still need to add you to that project
< jonasschnelli>
bitcoin/ should be consensus critical stuff
< wumpus>
jonasschnelli: we're not actually doing anything with it at the moment
< wumpus>
jonasschnelli: right
< jonasschnelli>
+1 ;-)
< cfields>
topic has strayed a bit :)
< wumpus>
yes
< sipa>
petertodd: ok, agreed
< petertodd>
sipa: thanks!
< wumpus>
#topic convert python RPC tests to python 3
< gmaxwell>
I am disappointed that all of this time is being wasted over faffing about libraries, for code that has very narrow independant interest and likely against the wishes of its author. .. and no one commented on further verification strategy.
< * gmaxwell>
sees topic is over.
< MarcoFalke>
I think it's cleaner to just switch to py3
< petertodd>
wumpus: concept ack!
< MarcoFalke>
but I'd like to hear any drawbacks first.
< wumpus>
gmaxwell: I agree, I think this will do just as well as a code snippet, but I don't think discussing this further is constructive. The people that want a library so bad can work on it I guess :)
< Luke-Jr>
py3 vs py2, I can think of no drawbacks to exclusively py3
< jonasschnelli>
Is the reason for the py3 switch the deprecating of py2 or the missing of it in some of the distributions?
< wumpus>
I think this makes a nice moment to decide on a switch
< MarcoFalke>
ok, so I will close the "refactor py2 code" pull and just go py3 only
< wumpus>
we're modernizing the build environment anyway by starting to rely on c++11
< wumpus>
MarcoFalke: it's the first step right
< jonasschnelli>
ACK on py3... the RPC tests are not something we need "full portability".
< wumpus>
for the build system I have py2+py3 compatibility ready
< Luke-Jr>
actually, we should retain support py2 at least initially
< Luke-Jr>
since we may need to backport
< wumpus>
for the RPC tests I don't think that is necessary
< petertodd>
I'm not sure there exist any build environments that we need to support that are py2 only
< wumpus>
agree petertodd
< jonasschnelli>
agree with petertodd
< cfields>
agreed
< sipa>
agree, the backport trees could also just switch to py3
< Luke-Jr>
if it's not much effort, moving to py2+py3, and then to py3 before 0.13 would be best
< wumpus>
Luke-Jr: that's a lot more work
< Luke-Jr>
if it's a lot more work, forget it
< wumpus>
I did that for scripts used by the build system itself
< wumpus>
but the test framework is huge
< wumpus>
and it's a lot of work for people to make sure it works for both python versions
< MarcoFalke>
considering that the current rpc test are full of bugs due to missing maintanance and review, doing a py2+3 coverage is impossible
< wumpus>
for every change
< sipa>
i think we should just outright switch to py3
< jonasschnelli>
Yes. No backward comp. for test script please.
< petertodd>
fwiw, I'm somewhat in favor of making my own python-bitcoinlib py3-only, simply because I personally never use it on py2, which makes me worried I'm missing bugs
< wumpus>
exactly, practical concerns kind of rule this out
< btcdrak>
I think we should switch to py3 outright also
< wumpus>
that's clear, then
< jonasschnelli>
And kudos to MarcoFalke for working on this!
< wumpus>
yes!
< btcdrak>
+1
< MarcoFalke>
You can pipe the files through 2to3 and then "only" check the diff.
< wumpus>
ok, that concludes the topic I guess, I'm happy we can finally agree on something :)
< cfields>
yes, it's much easier for us than most because it's all internal and we really have no downstreams to cater to
< jonasschnelli>
sipa: how would you support test if the ctaeslib is just a c./.h?
< jonasschnelli>
Add a test.c and a Makefile?
< gmaxwell>
with a define.
< gmaxwell>
#ifdef CTAES_TESTS int main()...
< sipa>
i preferna test.c
< sipa>
as it's perfectly externally testable
< Luke-Jr>
in this case, it may be wise to have a function for tests that software can call at startup
< gmaxwell>
sipa: k.
< Luke-Jr>
ie, refuse to start if the AES code fails its own testws
< gmaxwell>
well a test harness for verification may be very different than built in selftests.
< Luke-Jr>
sure
< petertodd>
test harness for the constant-time functionality is probably going to be a lot less portable than the code itself for instance...
< gmaxwell>
well for example, a verification test harness would likely contain another whole implementation of AES. :)
< cfields>
sipa: just pushed the code-movement to net-refactor8 branch. Still lots of placeholders, todos, and hacks in there, but it now looks overall how i want it to
< sipa>
gmaxwell: hmm... about tests; the s-box implementation is the most obscure/unreviewable/largest part of the code... but it's trivial to.test exhaustivelg against another implementation, as it only deals with 32-bit inputs
< sipa>
ah, nvm
< gmaxwell>
sipa: right. an exhaustive sbox test, e.g. with forward sbox pulled out of another implemetation. and checking forward and reverse. And then I thought it might be interesting to see if the existing vectors are enough to catch any simple or or two byte mutations in the current implementation. And if not, we could add vectors until they are.
< gmaxwell>
why cant the sbox be exaustively tested?
< sipa>
gmaxwell: not as a black box; its inputs are 8 16-bit integers (and.it does 16 sbox applications in parallel)... you can't exhaustive test 128 bits (i hope)
< gmaxwell>
right. well it could test each lane one at a time.
< sipa>
if you can inspect the code and see that it's "obviously" doing the same thing for every bit position, it suffices to test just the 2^8 inputs
< sipa>
petertodd: how is the bip9 warning code broken?
< sipa>
(re: github pr)
< petertodd>
sipa: oh, I spoke to soon on that - but it was more confusing than it should have been, which I'll fix next week when I get back after the long weekend
< sipa>
petertodd: the warning logic uses bit positions; the trigger code uses DeplymentPos
< sipa>
(i agree DeploymentId is better)
< petertodd>
sipa: yeah, simple enough pull-req, and I've got a few more improvements re: adding comments that I'll add to it too
< sipa>
great!
< petertodd>
oh, one thing I forgot to ask: if I understand it correctly, right now changes to the "meta-version" - the top three bits - do *not* trigger a "network may have upgraded warning"
< sipa>
petertodd: they do if it's permanent
< petertodd>
sipa: ah, alright, I'll go look at that more then and verify
< sipa>
there is the 50% of last blocks habe unexpected version warning still
< petertodd>
sipa: ah, ok, good
< sipa>
in addition, there is warning logic for each bip9 bit position, which works retroactively
< petertodd>
sipa: anyway, probably best I don't ask too many more questions to keep my mind uncontaminated with what you guys think the code should be doing :)
< cfields>
wumpus: i have some work on docker that may not have been in vain, though. It looks like a very possible gitian replacement
< warren>
cfields: for your own deterministic toolchain?
< wumpus>
cfields: interesting
< cfields>
(after all, Gitian basically is an early/simplified version of what Docker has come to be)
< wumpus>
in a way, yes. At least docker is very actively supported/maintained.
< warren>
docker still puts an OS into a chroot
< cfields>
warren: kinda. it just builds from a pre-determined sandbox and caches each step. as a bonus, each step can be shared as well
< warren>
IMHO it's worthwhile to switch away from gitian if we are at the stage of being fully in control of our own deterministic toolchain
< cfields>
warren: well you'll always need a reference environment
< warren>
cfields: the reference environment could be deterministic as well
< wumpus>
that's a long way, you don't just need a reference toolchain, but there's also other tools that have an effect on the result
< cfields>
in this case, docker basically acts as that reference environment, containing only our reference toolchain and the bare minimum other tools
< cfields>
seems to me, docker is the perfect tool for us in this case
< wumpus>
warren: we need a deterministically buildable OS, isn't debian working on that :)
< cfields>
warren: sure. and you have to run it somewhere :)
< warren>
ANY OS > build stage A that probably isn't deterministic -> use that to build stage B
< cfields>
warren: regardless of the determinism of the base, you have to have some tool for actually building the thing. Short of handing out a rootfs.tar.gz and praying that it will run on some baremetal, ofc
< wumpus>
anyhow this is driftng off topic
< wumpus>
main thing I'd like is to have c++11 in travis
< cfields>
warren: btw, I believe I've seen some early deterministic Debian base images on dockerhub
< wumpus>
cfields: so deterministic debian doesn't only exist in theory anymore? that's good to hear there's progress
< cfields>
wumpus: yes, i'm committed to making that happen one way or another. The net code depends on it, so I'm kinda forced to :)
< wumpus>
cfields: it sounds like a project akin to building the pyramids
< cfields>
hmm, trying to re-find what i saw
< wumpus>
even getting the bitcoin dependencies deterministic was a hell of a lot of work, just imagine for a full OS
< wumpus>
although it gets easier, with better tooling, some things only had to be done onece
< cfields>
wumpus: hmm, i might've made up the Debian part. I saw someone working on a deterministic base for Travis, my brain might've just lumped 'em together.
< cfields>
s/Travis/Docker/
< wumpus>
right - any deterministic base wouldb e fine, it doesn't need to be debian
< wumpus>
docker is based on lxc isn't it? that means it at least doesn't need deterministic kernel, init etc
< cfields>
based on cgroups, I don't believe it uses lxc itself
< wumpus>
right, the same underlying mechanism
< kanzure>
in addition to deterministic debian please also consider deterministic nixos things
< kanzure>
fwiw i recommend not using docker's build tool. it's a mess.
< kanzure>
using docker for running containers or something is probably fine (although coreos/rkt things should be considered, since they seem to be compatible with docker containers and have a specification and words written)
< kanzure>
"docker build" is a build tool crammed into a container management system. a more realistic build tool is something like bazel ( http://bazel.io/ ), which can spit out finished container images.
< GitHub178>
[bitcoin] jonasschnelli opened pull request #7742: [Wallet][RPC] add missing abandon status documentation (master...2016/03/ab_doc) https://github.com/bitcoin/bitcoin/pull/7742
< kanzure>
(with "docker build" you lose out on things like build caching unless you always build only on one machine. this negatively impacts things like distributed continuous integration build runners.)
< cfields>
kanzure: docker would only be used as a way to launch our own build process inside of a pre-determined environment
< warren>
kanzure: coreos supports docker, not the other way around
< kanzure>
i don't think i said otherwise.
< kanzure>
cfields: right but even the pre-determined environment has a build toolchain. and the default way when using docker is to use "docker build". which is what my above rant is about.
< kanzure>
oh i guess that is inconsistent with me mentioning rkt. my mention of rkt is unrelated to my complaints about "docker build".
< kanzure>
(rkt doesn't do container building)
< cfields>
kanzure: I'll read up on those things
< kanzure>
ok feel free to pester me. i have been reading docker source code and such.
< cfields>
great, thanks :)
< GitHub139>
[bitcoin] luke-jr closed pull request #7047: [WIP] Backports for 0.11.3 (updated to 93ca5a3) (0.11...backports-for-0.11.3) https://github.com/bitcoin/bitcoin/pull/7047
< GitHub149>
[bitcoin] luke-jr opened pull request #7743: [0.11] Important backports for 0.11.3 (updated to v0.12.0) (0.11...backports-for-0.11.3) https://github.com/bitcoin/bitcoin/pull/7743