< waiting2compile>
hi, could someone walk me a bit through running Bitcoin's tests? I tried following the instructions in the README.md, though I suspect there's some extra setup that needs to be done that's missing, since running the command just gives make errors and the like (pardon me, I'm new to this and am trying to fix one of the good first bugs :))
< sipa>
have you looked at build-unix.md (or windows/osx, based on your environement)
< sipa>
?
< waiting2compile>
Ah I had not, I'll try following the instructions there and see if I have any luck
< ken2812221_>
wumpus: Thanks, I'll check that out.
< wumpus>
time to update my build infrastructure so I can finally build the 0.17 branch in gitian
< wumpus>
starting with building the most recent lxc, 3.0.1 apparently
< Gnappuraz>
Hi, I was going through the bitcoin documentation but I can't find the rationale behind the fact of using the prevTx.scriptPubKey in the place of curTx.scriptSig when creating the signature
<@sipa>
wumpus: i'm going to clear the 'needs release notes' for all PRs merged before 0.17
<@sipa>
or are there things in the 0.16 branch that haven't been in a release yet?
<@sipa>
are we going to include qt arm builds in 0.17 releases?
<@sipa>
i went through the list of merged PRs, and added "Needs Release Notes" here and there where i felt it was useful
<@sipa>
some of these already have release notes, but i think it's useful to have such a list to compare with the notes when they are finished
< wumpus>
sipa: make ssense
< wumpus>
+s
< wumpus>
nothing has been merged after 0.16.2 that needs release notes
< wumpus>
gitian build--at least for linux--of 0.17 branch worked, phew
< wumpus>
going to test at least the ARM executables
< instagibbs>
anyone else getting github.com domain name res failure for git
< instagibbs>
oh sike, i just have no internet on the device
<@sipa>
i confirm your IRC client is on the internet
< instagibbs>
the burdens of running multiple computing devices...
< BlueMatt>
hmmm, MarcoFalke points out that it looks like we may be undercounting mempool memory usage by sizeof(CTransaction)*txcount
< gmaxwell>
BlueMatt: so fix it?
< BlueMatt>
oh, wait, nvm, I'm misreading it
< BlueMatt>
gmaxwell: I was hoping for someone to double-check me, but I found myself to be wrong faster :p
< gmaxwell>
fixing it also has the property of showing you that you're wrong fast. :P
< BlueMatt>
lol true
< jonasschnelli_>
hmm... manually added nodes with -connect have no service flags?!
< jonasschnelli>
If its unknown if the peer supports NODE_ENCRYPTION, I guess just trying and ^NODE_ENCRYPTION if failed seems accptable
< gmaxwell>
probably just try. maybe in a couple years we mandate encryption for connect and addnode (and add something for a connect that doesn't support it)
< jonasschnelli>
gmaxwell: you mean also trying on peers not signalling NODE_ENCRYPTION via the service flags (and eventually update addrman's service flags if failed to encrypt)?
< sipa>
i would only try it for things that signal encryption
< sipa>
if it doesn't work, drop the flag in your addrman and disconnect
< gmaxwell>
I was specifically talking about connect and addnode, where there are no 'flags' before we connect.
< sipa>
ah, makes sense
< sipa>
uh, that means trying twice :(
< gmaxwell>
if they don't support it.
< gmaxwell>
Do you see an alternative?
< gmaxwell>
We could immediately introduce flags to connect and addnode to say that crypto isn't in use.
< sipa>
-encaddnode -encconnect :)
< gmaxwell>
encryption should be the default.
< gmaxwell>
Ignoring the initial deployment catch22, I dont think there is much reason to even support connect/addnode to non-encryption supporting things...
< jonasschnelli>
indeed...
< sipa>
yeah
< sipa>
but you don't want existing addnode=IP lines in bitcoin.conf files suddenly fail
< jonasschnelli>
I think if -netconnection is enabled (which could be the default), addnode (and friends) should enforce encrypted peers
< gmaxwell>
sipa: Major versions can break things, thats what release notes are for.
< sipa>
or we could add encryption support, and a way to add flags to addnode/connect in one version
< jonasschnelli>
I think failing on addnodes that are non encrypted is probably a good thing
< sipa>
and then change the default to encryption in a later version
< jonasschnelli>
or that,.. yes
< gmaxwell>
we could make connect and addnode able to take an parameter e.g. connect=1.2.3.4$nocrypto
< gmaxwell>
yes, that would be okay.
< sipa>
that may be necessary regardless, yes
< sipa>
i wonder if we should call it encryption
< jonasschnelli>
enciphering? *duck*?
< gmaxwell>
connect=1.2.3.4$crypto connect=1.2.3.4$nocrypto ... and the default if no specified starts out no and gets switched later.
< sipa>
maybe it should just be "v2 protocol" (which happens to encrypt, but it's really a new non-backward compatible protocol encoding
< jonasschnelli>
Yes. That sounds good... I just fear scope creeping if we label it like that
< sipa>
okay
< sipa>
just an idea
< jonasschnelli>
"Ah. v2 protocol, why don't use change the inv that way",. etc.
< sipa>
but encryption isn't something that should be advertized really
< jonasschnelli>
But since it's a new protocol (kind-of a one time chance), those ideas are maybe welcome
< sipa>
or "v2 transport"
< sipa>
it's not really the protocol that changes, just the encoding
< jonasschnelli>
sipa: what do you mean with "advertised"?
< jonasschnelli>
sipa: Yes. "v2 transport" is more accurate
< sipa>
if you use the term 'encryption' to describe the feature there may be a false sense of security risk
< jonasschnelli>
Hm... you mean by not protecting from an active MITM?
< jonasschnelli>
I guess using the word encryption when it comes to the v2 transport would not be entirely wrong though
< sipa>
well encryption certainly protects against certain attacks, but not nearly all the ones that people think of when you say encryption :)
< gmaxwell>
Yea, it's more than encrypion, also encryption implies properties that it doesn't provide.
< gmaxwell>
e.g. the oppturnistic encryption does not prevent MITM.
< jonasschnelli>
It eventually does prevent from MITM because an MITM would be easy to detect, but it does not protect from MITM
< sipa>
not without somewhat deployed authentication
< jonasschnelli>
Yes
< gmaxwell>
in any case, we'll want to be able to provide arguments to addnode and connect later for auth keys.
< jonasschnelli>
With the "stealth handshake" (the very first 32byte message/key exchange), is there anything we should plan for in case we want to add something like RLWE to the handshake
< jonasschnelli>
+?
< gmaxwell>
If we did something like add rwle we could establish the secp256k1 handshake first and then inside that stream upgrade.
< jonasschnelli>
gmaxwell: wouldn't that partially break the "stealth" component (if we assume ecdh in secp256k1 is broken) since the inner 2nd handshake would probably require standard p2p message encryption?
< sipa>
jonasschnelli: not more than the current secp stealth component is broken by being sent in cleartest
< gmaxwell>
^ plus the 'stealth's is pretty weak, it's mostly just making harder to use dumb pattern matching to block.
< jonasschnelli>
We could allow additional dummy data in the encryption handshake as we do in the v2 message encoding protocol to make DPI harder
< gmaxwell>
hm. it really would be much easier if the initial handshake had a flag. e.g. [ecdh key][byte] it could still block dumb pattern matching by making the byte xored with the last byte of the pubkey.
< gmaxwell>
kinda irritating to add RLWE after the fact, sadly.
< jonasschnelli>
I think the change is already pretty huge,... I think adding more should be avoided
< jonasschnelli>
The tor argument "collect now, decrypt later" may not be applicable 1:1 to bitcoin
< jonasschnelli>
Especially as long as there are no private p2p extensions and an internal auth mechanism
< jonasschnelli>
IMO deploying RLWE together with auth could make sense
< sipa>
how much code is it?
< jonasschnelli>
Right now +1,563/-178 (incomplete, missing tests)
< gmaxwell>
RWLE is small, one sec.
< jonasschnelli>
I guess the new code is very critical. Must be review profound
< gmaxwell>
sipa: the ref implementation of newhope appears to be Total Physical Source Lines of Code (SLOC) = 1,347
< gmaxwell>
which includes some tests and stuff.
< gmaxwell>
obviously it's bigger with the AVX versions and whatnot.
< jonasschnelli>
I guess AVX for Chacha, Poly1305 and newhope could follow later?
< gmaxwell>
as far as security goes, the interfaces is really trivial, so it's easy to review that the worst risk it presents is not adding security, leaking something about its randomness, or being slow.
< gmaxwell>
It also has been deployed in _chrome_ as part of an expirement with ssl.
< gmaxwell>
(they made an expiremental handshake for SSL that did the H(ECDH||newhope) thing.
< gmaxwell>
I think the worst outcome from deploying it is that a month after wide deployment, the security is broken completely and we're stuck carrying it around (and wasting cpu cycles on it) even though it does nothing. :P
< jonasschnelli>
I somehow would prefer a two step implementation (and eventually also specification) approach from current v1 non encrypted network protocol to a quantum safe v2 (or v2.1) protocol
< gmaxwell>
ah, the torref/toravx implementations are apparently constant time.
< jonasschnelli>
Is there an anti DPI argument for using 32byte keyhandshakes rather then 64byte?
< gmaxwell>
jonasschnelli: sad thing is that the quantum safe thing is probably not worth doing on its own.
< gmaxwell>
jonasschnelli: what would the extra 64 bytes be?
< gmaxwell>
er extra 32.
< jonasschnelli>
if we want to add a flag but want to avoid 33bytes (or 34) due to DPI issues, we could pad up to 64?
< gmaxwell>
I don't think we want to avoid 33. We want to avoid fixed bytes. E.g. "if bytes 32-45 are 0xdeadbeef... reject"
< jonasschnelli>
Oh that. So the xoring with the last pubkey byte for the flag could then be accptable... I think
< gmaxwell>
right.
< jonasschnelli>
gmaxwell: why is upgrading the handshake later to include newhope be "not worth doing on its own"?
< jonasschnelli>
you mean the additional (questionable) security versus the deployment hassle?
< sipa>
i think upgrading the transport can be upgraded later easily enough that we don't need to rush including it right now
< gmaxwell>
because the security benefit is quite conjectural. so taking a network upgrade cycle, with incompatiblities and stuff, to maybe gain nothing.
< gmaxwell>
So for example I think to do newhope nicely later, just a flag isn't enough.
< gmaxwell>
because you want the initator to send their DH value in the first message.
< gmaxwell>
well I don't think we need to rush for sure. But if nothing else it's useful to think about how we would take the next step.
< gmaxwell>
So lemme talk though my thought process a bit.
< jonasschnelli>
what if the flag comes first to the message content and length can change later?
< gmaxwell>
earlier today I was thinking "we could deploy newhope by just brining up the secp256k1 encryption, then sending a rekey message that triggers upgrading." but then I realized that runs into the problem we had before of having to do the keying twice-- once for each direction.
< gmaxwell>
jonasschnelli: how does the recipent even know the length?
< gmaxwell>
(if its not fixed)
< sipa>
wahaha, the low-security version of newhope is called jarjar
< jonasschnelli>
gmaxwell: right now,... is just looks for 32bytes, if it matches a version message, it transforms the bytes into a legacy v1 message container
< jonasschnelli>
(and continues with legacy protocol)
< jonasschnelli>
if not a version message, it tried the handshake
< jonasschnelli>
*tries
< sipa>
that seems overly complicated
< gmaxwell>
okay, so your point is that there could be extra data, but how does a current client know to ignore the extradata? e.g. why won't that just look like an invalid encryption once the handshake completes.
< jonasschnelli>
sipa: idea how to make this simpler?
< sipa>
i think you should just assume a connection is encrypted if the flag is set, and it will fail if it turns out it wasn't encrypted
< sipa>
at which point you just disconnect and try another peer
< jonasschnelli>
sipa: I thought we want a mode where encryption is optional
< gmaxwell>
It is optional.
< gmaxwell>
sipa: how does this avoid an attack where the first peer you connect to gives you the encryption flag set for everyone, causing you to be unable to connect to most of the network?
< jonasschnelli>
I can't follow. That would mean we drop every connection that failes to do a handshake (think of SPV clients, etc.)?
< sipa>
jonasschnelli: ah, for incoming connections!
< sipa>
ugh
< gmaxwell>
I had the same confusion as sipa.
< jonasschnelli>
Outgoind is easy
< jonasschnelli>
Incomming IMHO must be ready to detect a handshake OR a version legacy msg
< sipa>
unless they're separate ports or something like that... but yeah
< jonasschnelli>
But code wise its simple: buffer the first 32bytes, check if it is (very likely) a version message and migrate the message type to a legacy message
< jonasschnelli>
(& avoid pubkeys that start with the network magic & 'version')
< sipa>
ah yes, that seems reasonable
< jonasschnelli>
If the handshake flag would be the first byte, and the xor key the second, we could read to bytes and figure out if we need to buffer 64 or 32
< sipa>
i think it could just be "pubkeys cannot start with the network magic"
< jonasschnelli>
Wouldn't that reduce the possible key-space by 2^28?
< jonasschnelli>
I guess I'm wrong though. :)
< sipa>
from 255 bits to 254.999999999664 bits of entropy
< gmaxwell>
and not in a useful way that helps attacks regardless.
< jonasschnelli>
Okay. I see
< jonasschnelli>
Is using the first pubkey bytes as flag xor key reasonable?
< sipa>
can't the flag be inside the encrypted stream?
< jonasschnelli>
It can, but we then would assume that ECDH must always be done
< sipa>
i think that's fine
< gmaxwell>
now we wind back to my prior comment about upgrading and synchronizing both directions.
< sipa>
i don't mean a flag to signal upgrading
< jonasschnelli>
I guess then we don't need a flag, we can handle it with messages then
< sipa>
just a flag that says "this is protocol version X"
< gmaxwell>
I think it's fine if ECDH is always done.
< sipa>
if you don't understand the flag, disconnect
< jonasschnelli>
gmaxwell: why is re-keying in both directions a problem?
< sipa>
and in a new version, the RWLE can be mandatory in both directions, with no synchronization issues
< gmaxwell>
twice the computation and overhead.
< jonasschnelli>
AFAIK the current rekeying can only be initiated by the encryption responder (server) by sending a "rekey" message
< sipa>
both parties should be able to rekey
< sipa>
or it should be automatic after a certain amount of data
< jonasschnelli>
Yes. 1GB is currently the specs and not below 10s
< gmaxwell>
sipa: I pointed out before that we ought to support time based so that we get forward compromise resistance on low traffic links.
< sipa>
also, is rekeying just hashing the existing encryption key, or is it a new ECDH negotiation?
< jonasschnelli>
just hashing
< gmaxwell>
it should be the former.
< gmaxwell>
right.
< sipa>
okay
< gmaxwell>
I don't see any value in repeating ECDH for incremental rekeying.
< sipa>
yeah
< gmaxwell>
in any case, if your intention is to encrypt the protocol version that means the initator cannot send a version without an extra roundtrip.
< sipa>
i don't think that matters
< gmaxwell>
I guess we don't really care too much about roundtrips, but it's the consequence.
< sipa>
the responder picks the version of the protocl
< sipa>
of the initiator doesn't like it, disconnect
< gmaxwell>
but the initator has to offer.
< gmaxwell>
or the responder will pick something the initator doesn't support, gratitiously.
< sipa>
that's no different than now
< gmaxwell>
So say, for example, you deploy newhope on your node... now you lose crypto to all existing peers? because you pick newhope and then they drop you?
< sipa>
ah yes, i see; there is an assymetry in knowledge
< sipa>
the initiator can be expected to know what the responder supports beforehand, but not the other way around
< jonasschnelli>
Would that be an argument for the flag at the very beginning of the handshake?
< gmaxwell>
or, it's just something that we'd have to handle with an additional round trip.
< sipa>
i think that's fine
< gmaxwell>
e.g. ecdh-> <-ecdh,enc(<flags>, extradata like a new hope handshake) ->enc(flags, payload)...
< jonasschnelli>
But I currently don't know how to handle the re-key sync problem,.. I guess dropping all messages after peer has sent the "rekey" message until it gots the "rekey-ack" response is a no go
< gmaxwell>
the rekey is determinstic. it should just be you send a rekey message and then the very next byte is encrypted with the next key.
< sipa>
and do it separately for both directions
< gmaxwell>
And have the rekey operate independantly for each direction.
< gmaxwell>
rekeys are cheap, they're fine to do independantly.
< jonasschnelli>
Ah yes. That would work.
< gmaxwell>
I want to avoid doing DH protocols independantly even for a newhope upgrade though.
< sipa>
there could also be a reqrekey message message, which requests that the other party do a rekey, but it has no effect on the protocol until they respond with a rekey
< gmaxwell>
sipa: I think instead you can just say that you have to do at least one rekey between two rekeys by the other side.
< jonasschnelli>
or could an initiated rekey from peer A requires an rekeying from peer B within a timeout of X?
< gmaxwell>
jonasschnelli: then you get into a rekey looop for a link with latency higher than X. :P :P
< gmaxwell>
But just saying that you must rekey if you have seen two rekeys in the other direction since the last time you rekeyed, I think doesn't have that problem.
< jonasschnelli>
heh... on dump implementations, yes. But point taken.
< gmaxwell>
or don't even worry about it and just specify that you must rekey after 1hr or 1GB (whichever comes first). And anyone who doesn't is non-conforming and if anyone cares, they could start banning based on the behavior.
< jonasschnelli>
Yes. Seems the be required anyways.
< jonasschnelli>
Okay. Thanks. Going back to the "drawing board" (specs and impl.).
< sipa>
i guess a rekey request is pointless; the other party could obey the request, but still keep the old encryption key in memory