< Jmabsd>
gmaxwell, is there any 256bit hash that is *not* printed in this order? also,
< gmaxwell>
I don't know what code would do that printing...
< Jmabsd>
i see the stronger, material, original reason for this ordering was to get the zero bits in block hashes have the zeroes in the beginning
< gmaxwell>
21:30:21 < gmaxwell> block hash is interpreted as a 256-bit little endian number for target comparison as luke says, so then bitcoin needed to print it that way too, otherwise the 0s would have been on the right which would have been confusing... and the same code was then used to print other hashes.
< Jmabsd>
however may there be some more general, common-sense reason too to print hashes in reverse order, or it's all just about symmetry with the block hash representation?
< Jmabsd>
ahaa, to get a proper ordering, interesting
< Jmabsd>
that makes sense yes. what code in Bitcoin Core compares two uint256:es, which would be used for sorting?
< Jmabsd>
HEX STRING -----(funny reverse order deserialization)----> uint256 structure which is actually a 32B byte vector ------(sort function, compares the uint256:es by normal < > logic) ----> sorted list ----- (serialize with funny reverse order again) ----> alphanumeric order sorted list
< Jmabsd>
that only makes sense if the < > operator is little endian :)
< Jmabsd>
ahh i see base_blob's < operator and Compare method
< Jmabsd>
the < logics are in int256.h's base_blob, and implemented as "friend inline bool operator<(const base_blob& a, const base_blob& b) { return a.Compare(b) < 0; }", "inline int Compare(const base_blob& other) const { return memcmp(data, other.data, sizeof(data)); }".
< Jmabsd>
memcmp here would react at the first byte that's different. this comparison logics are suitable for a *big endian* representation of a number right -
< dcousens>
Jmabsd at this point, we're simply stuck with it
< dcousens>
You can reason about it in a thousand ways, network byte ordering, etc, but, its merely what was done, and now its stuck
< Jmabsd>
i compare "0x12 0x54 0xfd 0x55" with "0x12 0x54 0xf0 0x55", in this case memcmp will give a negative value due to the third slot's difference
< Jmabsd>
dcousens: yeap i see. i'm trying to make just a bit of sense here, so that, if you make a Dcousens Protocol to do some Bitcoin-related stuff, and you have some structure in your protocol that you hash and then give to people, in what order should it be.
< Jmabsd>
for instance, for symmetry with Bitcoin, if you make a Dcousens Bitcoin Transaction format, then obviously its hash should be printed in reverse order.
< Jmabsd>
(which is some Bitcoin transaction for some purpose)
< Jmabsd>
e.g. Lightning channels certainly represent the channel open/close transactions as txid:s with reverse order, and maybe some other hashes too.
< Jmabsd>
dcousens: so the uint256 < operator is a big endian thing.
< Jmabsd>
wait where's the PoW bits counting done again - i guess it should count the *last* bits (at the last byte positions of the uint256 structure)?
< Jmabsd>
otherwise the PoW counting and the uint256 < operator are not symmetrical
< luke-jr>
it doesn't count bits at all
< Jmabsd>
or, wait
< Jmabsd>
they are
< Jmabsd>
er.
< Jmabsd>
luke-jr: where is the zero counting done again
< luke-jr>
it's not
< luke-jr>
"zero counting" is a simplification for end users
< luke-jr>
the protocol itself simply does a <
< dcousens>
personally, I use the terms `txhash` for the normal byte ordering, and `txid` for reverse byte ordering
< dcousens>
if they need access to the raw data (aka, hashing), they typically deal with the hash, if they are doing lookups, I give the reverse byte order hex string
< Jmabsd>
dcousens: and for blocks you call it what?
< dcousens>
blockhash blockid
< dcousens>
respectively
< Jmabsd>
dcousens: and if you ever print a 256bit or 160bit hash in a pubkeyscript, then you print it in *byte order* right
< Jmabsd>
dcousens: merkle root "id" versus "hash" too? :-))
< dcousens>
define print
< Jmabsd>
dcousens: "show to a user in a web page or console"
< dcousens>
yeah, in normal byte order for that
< dcousens>
again, I only make the distinction for block/tx id's
< Jmabsd>
dcousens: aha. Bitcoin Core would print the merkle root in reverse order. anyhow right.
< Jmabsd>
dcousens: i think your approach makes sense.
< dcousens>
and merkleroots*
< Jmabsd>
dcousens: you reverse-order your merkle roots too ???
< Jmabsd>
wait, hashes are such are binary/byte concepts so the whole idea of relating to them as comparable numbers, is a concept that Bitcoin adds on top of them, right? the inventor of SHA256 never related to hashing as SHA256(byte string) => 256bit integer which may be little or big endian, right?
< Jmabsd>
dcousens: aha. so "blockid", "txid" and "merkle root" you do reverse order. any more? HD wallet seeds?
< dcousens>
I'd say that is all
< Jmabsd>
(y)
< dcousens>
"blockid", "txid", and merkle roots... if I remember correctly.
< dcousens>
Typically, I refer to it as "merkleRootId"
< achow101>
Jmabsd: Bitcoin Core prints everything that is a byte array in reverse byte order
< achow101>
Jmabsd: except for byte arrays in scripts. those are printed as-is
< achow101>
literally everything else (block hash, merkle root, txid, tx hash, whatever) is reverse byte order
< Jmabsd>
achow101: if you print a transaction's body, that's in normal byte order no ??
< Jmabsd>
sorry for being spammy about this byte order question, in a way it's trivial however as soon as a user is trying to use this, if the ordering is wrong then things break.
< achow101>
Jmabsd: a transaction is an object, not a byte array
< dcousens>
achow101 except private keys
< Jmabsd>
achow101: right so Bitcoin Core defines both 256bit and 160bit hashes as subclasses of "base_blob", is this what you mean by byte array?
< achow101>
Jmabsd: yes
< Jmabsd>
achow101: is anything else than 256bit and 160bit hashes represented as "base_blob"??
< achow101>
no
< Jmabsd>
achow101: right. (what you say i get confirmed by "grep -r base_blob *" too - it's only mentioned in uint256.*.)
< achow101>
dcousens: private keys are printed as WIF :)
< Jmabsd>
just for reference, where is the comparison of PoW, that is against the difficulty target, and in a reorg?
< Jmabsd>
that should be a "<" operator on the uint256_t type no?
< achow101>
Jmabsd: src/pow.*
< dcousens>
achow101: my point was, in terms of a community standard
< dcousens>
the standard is reverse byte order for blockid, txid, merklerootid ... I don't think there is anything else you would want to adhere to in reverse byte order.
< dcousens>
And therefore, even when displaying, I would _only_ do reverse byte order for those things, e.g, if you are printing a private key, you wouldn't print it in reveres byte order
< achow101>
dcousens: it's actually anything that can be interpreted as an integer. e.g. version numbers are little endian which is a form of "reverse byte order"
< dcousens>
I don't think I've seen the locktime printed as reverse byte
< dcousens>
But maybe!
< achow101>
dcousens: is it not reversed from whatever it is stored as?
< dcousens>
it is stored in LE
< dcousens>
so not reversed afaik
< achow101>
right, so it is printed as a big endian decimal which is reverse byte order of little endian
< dcousens>
ha
< dcousens>
yeah I suppose
< dcousens>
but if I saw it in hex
< dcousens>
I'd probably assume LE, not BE
< dcousens>
unless I saw `0x...` haha, then I'd assume BE
< dcousens>
what a mess
< Jmabsd>
wait, for mainnet, configuration includes: consensus.powLimit = uint256S("00000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff");
< Jmabsd>
this means that a single proof of work may not.. what?
< fanquake>
cfields are you doing detached sigs now?
< Jmabsd>
dcousens,achow101: facepalm lol.
< Jmabsd>
dcousens,achow101: i like dcousens' way of relating to it that only blockid, transactionid and merklerootid should be reverse byte order, that limits the possible damage from ambiguous order conventions too ha.
< Jmabsd>
achow101: the "nBits" value, which is copied from the block header and is applied in PoW here https://github.com/bitcoin/bitcoin/blob/master/src/pow.cpp#L80 : "arith_uint256 bnTarget;bnTarget.SetCompact(nBits, &fNegative, &fOverflow);" , it's in the Bitcoin-internal 32bit floating point format right??
< achow101>
Jmabsd: yes
< Jmabsd>
achow101: actually what code generates that bits value??
< Jmabsd>
achow101: at the genesis block, the bits value is *very low* and then it's getting *higher and higher*, slowly approaching 2^256 right?
< Jmabsd>
achow101: each PoW generated out there, has the block hash proof of work 256bit-integer-compared with this target value right?
< gmaxwell>
it goes up and down, whatever is required to keep the block rate on target.
< Jmabsd>
so LibBitcoin is checking that... err.. the target is.. BIGGER than the PoW?
< Jmabsd>
gmaxwell ah yes.
< Jmabsd>
LibBitcoin comments "Ensure actual work is at least claimed amount (smaller is more work)." - i wonder how they got this inverse definition of work, he. thoughts?
< achow101>
Jmabsd: Bitcoin Core checks that if the block hash is bigger than the target, it should fail, hence the return false that follows
< Jmabsd>
Ah. yes.
< achow101>
libbitcoin checks that the hash is less than the target, and if it is, it continues
< achow101>
they mean the same thing
< Jmabsd>
achow101: noted. and... what's the point here?
< gmaxwell>
I've been asking myself that.
< Jmabsd>
you want the work to be SMALLER than the target? so the "bits" value is going to zero??
< Jmabsd>
so zero means 256 leading zero bits to qualify? :-}
< achow101>
Jmabsd: yes. the hash should be less than the target
< Jmabsd>
achow101: this makes sense if "leading zero bits" means "the most significant bits in the block hash viewed as a 256bit unsigned integer" -
< Jmabsd>
then obviously zero bits means a SMALLER integer
< Jmabsd>
ok interesting, this is a very curious piece of endianness juggling, ha.
< achow101>
Jmabsd: the whole target checking works like this:
< achow101>
interpret the hash (as output by the hash function) as a 256 bit little endian integer
< achow101>
take nBits, decompress it and interpret it as a 256 bit little endian integer
< achow101>
if the hash is less than or equal to the decompressed nBits, then it is valid.
< achow101>
The whole reverse byte order thing is to get "256 bit little endian integer" into "256 big endian integer" because humans are used to looking at things in big endian
< Jmabsd>
right. (and, nBits is a custom value format so "little endian integer" here doesn't make sense as such. point is just, "make a 256bit integer type so you can compare it, and don't mess up!" he.)
< achow101>
yep
< Jmabsd>
achow101: and then for sorting blockid/txid/merklerootid:s, you want the internal 256bit integer sorting to correspond to sorting the string serialization in big endian representation form lol
< gmaxwell>
there isn't any normative sorting in the protocol.
< Jmabsd>
mhm.
< Jmabsd>
so we can only understand that transaction id:s and merkle root id:s (and in Bitcoin Core's case also hd wallet seeds) got this reverse byte order,
< Jmabsd>
was just a coincidence of how Satoshi happened to write the code
< gmaxwell>
they didn't "get" any reverse order.
< Jmabsd>
at least we can't see more sense in it today.
< Jmabsd>
gmaxwell, well in the human readable string representation, they do, right
< gmaxwell>
everywhere inside the system there isn't any reversing. It's just printed that way.
< gmaxwell>
yes, as was explained to you a day ago.
< gmaxwell>
Because blockhashes make sense to print in that order, every other 'big number' just used the same printing code.
< Jmabsd>
yeap exactly - the convention of printing transaction id:s and merkle root id:s in reverse order, we can only understand it to be a coincidental design choice that originated from printing block id:s in reverse order, which does have sense in that you get the zeroes in the beginning that way, which is educative.
< Jmabsd>
exactly. ok i think i got it now. regarding dcousens' terminology of "block id" vs "block hash", "tx id" vs "tx hash", and "merkle root id" vs "merkle root hash", is this a widely accepted convention, or is it just his thing??
< Jmabsd>
where "id" would mean reverse order and "hash" would be normal order
< gmaxwell>
I'm not familar with it.
< gmaxwell>
Really, if byte orders are tripping you up you're going to be really confused by every other detail of the system. :)
< Jmabsd>
lolol. no i just look at this with particular attention as any user who would get his data trashed would get really confused, ha
< Jmabsd>
i mean, say someone makes a program asking a user to pass a "block hash" as an argument
< Jmabsd>
and the guy gives it in reverse order, and then the program says "no such block exists, bye"
< Jmabsd>
that kind of consequental malbehavior
< Jmabsd>
what about "id" versus "hash", is this an established convention re block/tx/merkleroot?
< achow101>
Jmabsd: fun fact, bitcoin core does exactly that
< Jmabsd>
achow101: :o where?
< achow101>
if you give it the actual block hash, it will not find the block
< gmaxwell>
achow just means that the input is in the same order as the output.
< gmaxwell>
21:36:50 < Jmabsd> what about "id" versus "hash", > 21:34:51 < gmaxwell> I'm not familar with it.
< Jmabsd>
achow101: derp derp. exactly.
< Jmabsd>
> achow just means that the input is in the same order as the output.
< Jmabsd>
what do you mean?
< gmaxwell>
what was was trying to say is that the order bytes are in are the most trivial of 101 things you have to get right for software to work. I don't really spend any time thinking about it. Sometimes I'll do something and it won't work because I needed to reorder something, so I do, and I move on.
< achow101>
Jmabsd: you give it the block hash in the way that it is printed, aka reverse byte order. if you give it how it actually is, it will not find the block
< Jmabsd>
achow101: exactly. so all over the whole Bitcoin ecosystem, those three values (block HASH, transaction HASH and merkle root HASH aka ID whatever), when you're dealing with hex representation, it must ALWAYS be in reverse order bc otherwise you'll screw up
< gmaxwell>
achow101: there is no spoon^w"how it is". I mean, it "is" electrical charges, but you enter it via moving buttons. Everything we deal with is an arbritary representation.
< Jmabsd>
so if you have a tool / app / web site whatever that tells you which block, then you give it in reverse order
< Jmabsd>
if you have a CLI tool / app / whatever that asks which block or tx, then you input it in reverse order, etc.
< achow101>
Jmabsd: yes
< Jmabsd>
however aside from block hashes, transaction hashes and merkle roots, you're free to use the normal order.
< achow101>
Jmabsd: no
< Jmabsd>
achow101: no? :o
< achow101>
Jmabsd: those still use reverse byte order
< Jmabsd>
yea that's what i said:
< achow101>
oh, I misread, nvm
< Jmabsd>
achow101: if anywhere in this world you show a user a block hash, tx hash or merkle root hash, or ask him for it, then you do it in reverse order - but, for other values, such as for instance if you make a HD wallet and print him a hex dump of a seed, or,
< gmaxwell>
hash160s are probably also in that order in the interface, if any show up in the interface anywhere.
< Jmabsd>
some other hash, even if it relates to the bitcoin protocol, then normal order is fine
< Jmabsd>
for instance, the 32B and 20B hashes in pubkeyscripts! those would be normal order
< achow101>
gmaxwell: it depends on where the hash160 is, apparently
< gmaxwell>
pubkeyscripts is not an arith/hash_n, it's just bytes.
< Jmabsd>
gmaxwell: exactly, Bitcoin Core's HDwallet seeds are 160bit and presented in reverse order, yes
< gmaxwell>
in any case, trying to come up with a greater theme is probably a waste of time. You do whatever works.
< achow101>
if it is being output in, e.g. getaddressinfo it will be in reverse byte order. but in decodescript, it is not
< gmaxwell>
There is a historical reason that it works the way it does, sure, but all that matters is that you do what works.
< Jmabsd>
achow101,gmaxwell: maybe to help the world a bit, it's better actually to talk about "block id", "transaction id" and "merkle root id", and by "id" you mean "reverse order", even though it's not universally established nomenclature - tihs was a nice suggestion by dcousens.
< gmaxwell>
if you want to predict what something will be (why?) then I believe "is it stored as a big uint type in the system" is the necessary and sufficient predictor.
< gmaxwell>
I think if you're spening enough time thinking about this that you need names for it, you're probably doomed to never do anything useful.
< Jmabsd>
achow101: actually when do you ever show a user the merkle root ID...
< Jmabsd>
the merkle root 256bit value is soo internal
< Jmabsd>
you never compare it with anything else too
< achow101>
Jmabsd: in getblock
< gmaxwell>
I can tell you that it's possible to work for years on the system and never think about any of this for more than a moment.
< Jmabsd>
achow101: you make a lookup of a block based on its merkle root or what :o
< achow101>
Jmabsd: nope, just as output for a decoded block
< Jmabsd>
right. and it's reverse order. derp derp. yeap.
< gmaxwell>
and I've seen from years in this channel that people that come in asking lots of questions about byte order almost all just get frustrated and give up.
< Jmabsd>
okok i got it. anyhow all clear now. thx.
< gmaxwell>
the order it prints big uints is arbritary, but no more or less arbritary than the decision to print them in hex instead of base2 or base64 or something else.
< gmaxwell>
the order made sense for one thing, and then everything else just did the same because they share printing code and the order was irrelevant for everything else.
< Jmabsd>
achow101: if you ever printed a merkle *node* to a user, would you do it in reverse order, i guess *not*.
< gmaxwell>
of course.
< gmaxwell>
if its represented as a uint in the software, it gets printed in that order.
< gmaxwell>
so it would be 'revese' if you mean printing an interior node hash.
< gmaxwell>
er. reverse.
< achow101>
Jmabsd: it would be in reverse because I would use a uint256 and that prints in reverse :p
< achow101>
(in a modification of Bitcoin Core)
< gmaxwell>
there are already a zillion random fields that show up in the UI in reverse order that litterally no one ever thought about. They just printed a uint and thats how the uints print method works.
< gmaxwell>
So, for example, there is something that prints a hash of the utxo set.
< gmaxwell>
that was added last year.
< gmaxwell>
Without looking I'm sure it's in "reverse" order. No one ever even thought about it. Because the order is irrelevant, so they just get the default behavior.
< Jmabsd>
gmaxwell, a hash of all utxo:s in the blockchain at a given block height?
< gmaxwell>
at the current height.
< gmaxwell>
it's useful for checking for corruption in the utxo set.
< Jmabsd>
derp derp, ok Bitcoin Core gonna kick in more reverse order hex conventions in the ecosystem over time, he he. however users won't care really about those.
< Jmabsd>
aha yep.
< Jmabsd>
one could call it "utxo set id" to subtly emphasise that it's reverse order.
< gmaxwell>
or, one could not care because it's a total waste of time.
< gmaxwell>
no one ever had to think about it at all.
< gmaxwell>
if you get too busy thinking about minutia you'll probably find your mind becomes too crowded to think about more complicated things. :)
< provoostenator>
The SDK file isn't distributed, right?
< jonasschnelli>
looks like... I though we have all shared the same tarball? But maybe not
< provoostenator>
I used OSX to extract the relevant stuff from the thing I downloaded from Apple. On Linux you have to use different tools. Or maybe it's a timestamp thing?
< provoostenator>
I've had the same hash since 0.16.0rc4 it seems, and I've used at least two distinct gitian machines.
< luke-jr>
jonasschnelli: the tarball isn't deterministically made..
< luke-jr>
IMO it's a good thing that we have obviously-different SDKs ;)
< jonasschnelli>
Yes. Indeed
< gnappuraz>
Hi there, I was wondering if there is any documentation relative to the different archives declared in the main Makefile. Cause I don't really understand the logic of how stuff is organized. i.e. what is the difference between libbitcoin_util and libbitcoin_common?
< jonasschnelli>
gnappuraz: these are semi-independent modules. Those modules get linked for the different binaries we have (bitcoin-tx, bitcoin-cli, bitcoind, bitcoin-qt).
< jonasschnelli>
But they do not have an ABI or are meant to be used with other applications
< ossifrage>
g++ is really annoying sometimes, a 'make -j4' just managed to OOM this crappy box
< gnappuraz>
thx for the answer. But let's assume that I have to add a new file that if meant to be use by both bitcoind and bitcoin-qt, the rationale behind putting it in COMMON vs UTIL should be: if used by bitcoin-cli then UTIL, otherwise COMMON (since bitcoin-cli links UTIL but not COMMON)?
< Empact>
Is it possible to extend the ability to restart appveyor builds to core devs?
< Empact>
ken2812221: Thanks - set up the former but don't have access to the latter.
< ken2812221>
Empact: I was talking to MarcoFalke, he's the owner of appveyor ci account.
< _flow_>
What can I do to get https://github.com/bitcoin/bitcoin/pull/13621 merged? It improves the, currently terriable, configuration mechanics of bitcoind a bit (while still not perfect) and re-enables a recently disabled test.
< wumpus>
we're really careful merging changes to the order of configuration file parsing, slight changes to precedence will mess up things for current users and there are no good tests for this
< jonasschnelli>
_flow_: it needs more developers weighting in on the importance of that PR
< wumpus>
_flow_: so I suppose my answer is "add good tests"!
< wumpus>
those would also help to show what is improved, as they would fail on the old code and pass on the new one
< wumpus>
also: does this affect the GUI
< wumpus>
I see you did update the existing test, good
< wumpus>
but I think it needs to extended, to include the failure cases that you describe
< jonasschnelli>
how do I compile to detect pass-the-end iterator violations? I once had this setup but can no longer reproduce it.
< jonasschnelli>
(ideally gcc/debian)
< jonasschnelli>
Oh.. is it maybe DEBUG=1 for the depends build?
< achow101>
gmaxwell: luke-jr: there's an additional "absurdly high fee" that is enforced as a relay policy. it's triggered by a tx with a fee higher than 0.01 BTC (or something like that)
< achow101>
gmaxwell: also, insanely high fee is something that I can only find in 0.8 and earlier. I think that guy is making his own altcoin from 0.8.6 as people tend to do
< treyzania>
why do people fork from that far back?
< cfields>
fanquake: I was waiting to make sure that my results matched jonasschnelli's before signing
< cfields>
jonasschnelli: yes, it's DEBUG=1 for depends
< cfields>
jonasschnelli: for sigs, note that this is the first detached sig in a new release series. So we'll switch to a 0.17 branch to match.
< jonasschnelli>
cfields: ack. I'll PR after you pushed your commit (if you havent)
< cfields>
jonasschnelli: pushed
< cfields>
jonasschnelli: for future reference, I use "--orphan" when creating new git branches in the detached sigs repo, to avoid sharing any history between them.
< jonasschnelli>
cfields: noted
< gmaxwell>
achow101: no there isn't. re: relay policy.
< gmaxwell>
achow101: you're probably confused because the check is in ATMP, but it's always called with it disabled when handlign a transaction from the network.
< wumpus>
what's up with travis, btw? it looks like it's failing on all PRs
< sipa>
feature_block fails?
< wumpus>
there's something wrong with this new appveyor build
< sipa>
oh, it's appveyor that fails, not travis
< wumpus>
"Error making request with Error Code ServiceUnavailable and Http Status Code ServiceUnavailable. No further error information was returned by the service."
< wumpus>
I've also seen feature_block fail though, but not consistently
< wumpus>
on the very-last test (the reorg)
< MarcoFalke>
ken2812221: Empact: I set the permissions to view and run builds for our github team
< MarcoFalke>
Could someone verify that?
< Empact>
MarcoFalke: I'm still not seeing an option to restart the appveyor
< MarcoFalke>
You need to be in the group that can assign labels to issues on our GitHub, I can't see who is in there
< Empact>
jonasshnelli: First three commits look independent as prep work
< jonasschnelli>
It a great concept to bypass DPI, censored envs...
< * sipa>
suggests using gramtropy generated text over IRC
< jonasschnelli>
Tor can start subprocesses that bootstrap a SOCK proxy to communicate with the counterparty
< gmaxwell>
jonasschnelli: Why do you think we need something special for that? We have proxy support already, which is all that is generally required for that sort of thing.
< jonasschnelli>
I think supporting PT should be not to complicated in Core
< gmaxwell>
I think it's so not complicated that we already have it. :)
< wumpus>
I don't understand
< jonasschnelli>
I'm unsure about the reverse proxy, the env vars and the auto-exec of that subprocess...
< wumpus>
pluggable transports plug into *tor*
< sipa>
jonasschnelli: what specific form of transport do you suggest?
< wumpus>
bitcoin core can use tor
< wumpus>
what else is needed?
< jonasschnelli>
sipa: its not about specific transports,.. IMO its an API for other transport layers...
< sipa>
jonasschnelli: it is to make tor communicate with other tor nodes using different transport layers
< sipa>
what transport layer do you suggest?
< gmaxwell>
The API is socks.
< wumpus>
looks like a layer violation to worry about this? or do you want to make tor tunnel over bitcoin instead of the other way around?
< luke-jr>
I also don't get it
< jonasschnelli>
Maybe I got it wrong,.. but it looked to me after a pure transport layer (not tor)
< sipa>
yes and no
< wumpus>
yes, socks (+optional control port) is good enough as API
< wumpus>
it's the only way it's recommended to use tor
< sipa>
jonasschnelli: i'm very confused by what you suggest
< sipa>
so far you've said "we can use tor pluggable transport", which is true for any tor user
< phantomcircuit>
jonasschnelli, the pluggable transport presents a socks interface which tor uses instead of directly connecting
< sipa>
how is it bitcoin specific, or what do you specifically propose?
< wumpus>
integrating tor's code into bitcoin core is not a good idea
< kanzure>
merge tor source tree
< phantomcircuit>
it's not something that should be configured by things using tor
< wumpus>
indeed, it's part of the tor configuration
< jonasschnelli>
The idea is to make Bitcoin work in DPI env in case it would get blocked,.. like China or Iran
< jonasschnelli>
Without relying on tor
< gmaxwell>
wumpus: I think jonasschnelli suggests that we have support for these obfscuated transports without putting tor in the middle. I think we already do (in fact, I used one of the ones made for tor to bridge bitcoin nodes last year when we were worried about china blocking bitcoin... before later changing to an icecast stream so that it wouldn't be identifyable by traffic analysis)
< jonasschnelli>
Core could directly use obfs4
< gmaxwell>
jonasschnelli: we already can.
< jonasschnelli>
or meek
< sipa>
jonasschnelli: ah!
< wumpus>
gmaxwell: ohh, so using *part* of tor, just the obfuscation part...
< sipa>
you mean using obfuscation for our own connections, even non tor ones
< jonasschnelli>
Since its a separated project and a seperate binary
< gmaxwell>
wumpus: well the obfuscation parts of tor aren't parts of tor, technicaly. But yes.
< wumpus>
this sounds like scope creep to me
< gmaxwell>
jonasschnelli: have you trie using obfs4. I did previously, it just worked.
< gmaxwell>
tried*
< sipa>
i would say if we're worried about that, you should also just use tor already
< wumpus>
gmaxwell: it's tor project code
< wumpus>
maybe not part of the tor repository, I don't know
< phantomcircuit>
i dont see any reason to do this, anybody who needs this can already configure it with the socks proxy support
< wumpus>
yes
< jonasschnelli>
okay... fair points. /topic
< gmaxwell>
AFAIK, it just works. It's kinda limited though because none of them resist traffic analysis and it's SUPER easy to identify bitcoin traffic with traffic analysis. :)
< phantomcircuit>
gmaxwell, iirc they're all super easy to identify anyways
< gmaxwell>
If there was some addon we needed to make more of them work, I suppose that would be cool, but I dunno what that would be. At least the original obfs proxy that I used before was just a socks proxy.
< gmaxwell>
phantomcircuit: well I think the idea in tor is that there are private ones.
< jonasschnelli>
I think core <-> core could use the propose p2p encryption (with NewHope), but for censoring, is was unsure if we should relay on Tor or on the underlaying independent obfuscation projects.
< wumpus>
oh the obfs layer itself is also a socks proxy?
< gmaxwell>
wumpus: yes
< wumpus>
tor is socks proxies all the way down and nothing else isn't it :)
< sipa>
pigeons at the bottom of the stack
< wumpus>
anyhow, yes, then we can already use it
< wumpus>
sipa: pigeons for transaction broadcasting
< phantomcircuit>
gmaxwell, the only ones that work are the domain fronting ones, so presumably only the private ones actually work
< wumpus>
well the non-private ones become known very soon
< gmaxwell>
(hm the thing I used before wasn't obfs4 it was an earlier version, they all look like they implement proxies regardless)
< wumpus>
no matter how good the obfuscation is, if it's public that you run one...
< gmaxwell>
In any case, I think jonasschnelli's spirt is right. I just don't actually think we need to do anything more than have socks support.
< wumpus>
yes
< jonasschnelli>
Even better then. :)
< gmaxwell>
(it's certantly better to do protocol obfscuation via external proxies than to try to bake anything in...)
< sipa>
how does the proxy know which peers need to use the obfuscation?
< phantomcircuit>
gmaxwell, meek is the only one that works in china, it's domain fronting with gce
< sipa>
you can't just blanket obfuscate all connections
< sipa>
as our normal peer discovery will find peers that don't support it
< achow101>
sipa: I think this would only work for outgoing connections
< wumpus>
sipa: you wouldn't want to use normal peer discovery in such cases
< sipa>
achow101: yes, of course
< sipa>
wumpus: right
< wumpus>
something needs to de-obfuscate incoming connections too, but I suppose that works by forwarding a listening port?
< gmaxwell>
I think perhaps the useful 'feature' that might come for this is being able to provide per-peer proxy configs to addnode/connect.
< wumpus>
or does it really need SOCKS5 binding
< jonasschnelli>
I think achow101 has a topic
< wumpus>
(which we don't support, we only use the proxy for outgoing, SOCKS binding is really really obscure)
< gmaxwell>
I don't think there is need for socks5 binding.
< wumpus>
okay
< wumpus>
per-peer proxies could be useful, yes
< cfields>
gmaxwell: interesting. I believe that'd be trivial to add, other than the syntax handling
< gmaxwell>
sipa: don't want to go too far or proxychains becomes the right tool (it's a multiproxy routing daemon)
< sipa>
ha
< sipa>
that sounds like an improvements over sidechains or treechains
< phantomcircuit>
gmaxwell, proxychains is never the right tool >.>
< gmaxwell>
(it can even nest proxies)
< gmaxwell>
phantomcircuit: but how else will I hax you through 7 proxies?
< wumpus>
fwiw ssh has built-in support for proxy chaining in the client
< wumpus>
#topic hardware wallet support (achow101)
< achow101>
So with PSBT we can kind of do hardware wallet things
< gmaxwell>
in any case, being able to net-scope or node scope proxies is something I would have used before, e.g. "addnode my node in the office via my ssh -D socks proxy".
< achow101>
I've been working on a tool that takes a psbt and does the hardware wallet things: https://github.com/achow101/HWI
< jonasschnelli>
great work achow101!
< gmaxwell>
achow101: cool.
< wumpus>
woohoo!
< achow101>
but to actually get hww support, we still need a way to get derivation paths for imported keys and such. and for those keys to be part of the keypool
< jonasschnelli>
why not a xpub watch-only wallet support?
< achow101>
I know sipa has the output descriptors stuff that would work for this, but I think that still quite a ways off
< sipa>
i can prioritize it more, but it's unclear how to keep things compatible
< sipa>
do we want the old keystore based system and the new descriptor based system side by side?
< jonasschnelli>
importing keys seems meh-ish for hardware-wallet. IMO the ideal use case for BIP32 pub key derivation
< sipa>
that wouldn't be too hard, i think
< gmaxwell>
I don't see why we shouldn't deploy the very narrow thing we need to make all the common hardware wallets work, so long as it won't prevent us from using descriptor based logic in tuhe future.
< sipa>
but it would be much nicer if we also get to convert old keystores to desceiptors
< jonasschnelli>
Also, xpub watch-only do not need keypools
< gmaxwell>
jonasschnelli: yes it does.
< jonasschnelli>
I don't see why but happy to learn...
< sipa>
it's the equivalent of their gap limit
< gmaxwell>
It doesn't have private key material, but it needs _something_ to look ahead with pubkeys and watch for payments.
< achow101>
I did #14021 for importing keypaths into the wallet with importmulti, but it isn't ideal
< achow101>
it ... works, but kinda sucks as you have to import more keys as you use them. And they aren't in the keypool
< jonasschnelli>
gmaxwell: yes. Right. Correction: no keypool written to disk, only xpub, then derive on load
< sipa>
i fear for more combinations and special cases to deal with
< gmaxwell>
jonasschnelli: well it doesn't really matter if we save it to disk or not. It's harmless to do so.
< jonasschnelli>
Yes. Thats right.
< gmaxwell>
achow101: obviously as a stopgap importing keys at least allows development and/or testing.
< gmaxwell>
achow101: but I assume what you really want is a wallet whos master key reflects the third-party keychain. I fear the mess of mixed wallets.
< jonasschnelli>
achow101: do you have a plan to make hww interaction more hassle free, like including USBHID in core </joke>?
< achow101>
jonasschnelli: the idea I had was to use the HWI scripts and call them from Core.
< achow101>
they implement all of the usb stuff
< gmaxwell>
I think that should be out of of scope. We should implement a simple fork and stdio interface (or similar) to talk to hardware specific drivers.
< jonasschnelli>
could one of the "hardware-wallets" just be a hot keystore? Similar to SSH Agent?
< gmaxwell>
right.
< sipa>
have metadata in the wallet that says "for these outputs, attempt to sign through device driver X"
< sipa>
which is just an executable that gets invoked and sent a psbt
< gmaxwell>
Even one using SGX+TPM, for example.
< gmaxwell>
the drivers could then also provide their own UI, if required.
< gmaxwell>
E.g. to get a passphrase.
< gmaxwell>
or remind you to plug in the device.
< achow101>
yup. I plan on adding a UI to HWI once I get everything actually working
< jonasschnelli>
Yes. Becaue of the UI, I initially though every vendor must provide its own script/binary...
< sipa>
jonasschnelli: yes
< jonasschnelli>
Because again, plugins then endup outside of the vendors control
< jonasschnelli>
Leading to outdated software
< gmaxwell>
I'm sure it would eventually be possible to have "generic" drivers that work for a family of devices, but I think the industry is too immature for that.
< jonasschnelli>
Indeed
< gmaxwell>
sipa: how would we handle, e.g. "this script is multisig, with device X and device Y" ?
< gmaxwell>
(er driver x and driver y)
< sipa>
gmaxwell: list two drivers
< gmaxwell>
sounds good.
< jonasschnelli>
I was hoping for a protocol/API where Core starts a process and passes all relevant data and the vendors plugin there with rich UI "drivers"
< sipa>
gmaxwell: the drivers would be metadata to descriptor records
< achow101>
gmaxwell: wouldn't that be "this script has two keys and key 1 uses driver X and key 2 uses driver Y"
< sipa>
so they would apply to all outputs derived using the same chain(s)
< sipa>
achow101: doesn't even need to say which key is which device
< sipa>
just "try signing using X, try signing using ay"
< sipa>
*Y
< achow101>
anyways, the main point of bringing up this topic was that I wanted to ask whether we should implement xpub watch only wallets or just wait for sipa to finish output descriptor wallets
< gmaxwell>
This kind of interface could easily be used for things other than HWI. For example, green address has a 2fa service, where the service will sign its part of a multisig with you, if you pass a 2fa. This could be supported by a simple driver that sends the tx to a service and gets the sig back.
< gmaxwell>
jonasschnelli: I don't think we should be providing the HWI specific UI, because if we provide the UI we would constrain innovation there.
< jonasschnelli>
Not Core. But the vendor could have a driver with UI
< jonasschnelli>
(and network interface and whatnot)
< gmaxwell>
right. I agree.
< gmaxwell>
The interface between core and the driver is basically just a PSBT plus arguments provided in the driver config.
< gmaxwell>
achow101: well I think thats between you and sipa mostly
< gmaxwell>
I hope we don't end up with even more combinitorial explosion in the wallet, however.
< jonasschnelli>
Yes. But IMO that only one part. How to get pubkey&metdata into core (and eventually multisig stuff) is still unlcear to me
< gmaxwell>
like having to handle a wallet with mixed old plain keys, hd keys, achowpubkey, and descriptors all at once.
< jonasschnelli>
(in a non RPCish way)
< sipa>
jonasschnelli: descriptors :)
< jonasschnelli>
sipa: I mean plug-n-play. :)
< sipa>
jonasschnelli: i'll try to create a writeup on how to integrate them in the wallet
< sipa>
jonasschnelli: "import a descriptor"
< gmaxwell>
the descriptor thing could eventually be "create new wallet with descriptor" "paste in descriptor" "confirm"..
< sipa>
ah, you mean GUI
< sipa>
i have no clue.
< jonasschnelli>
You plugin a HWW, the Core detects it and tells you "do you wanna use XYZ".
< jonasschnelli>
*then
< gmaxwell>
doubt we're going to have any detection anytime soon...
< gmaxwell>
as that would require shipping a pile of vendor specific detection and interface code.
< jonasschnelli>
Yes. But if we form an API or similar, we should not only focus on the PSBT part,... also on the how do I create a watch-only wallet (eventually also multisig)
< jonasschnelli>
But I guess the descriptors are the answer here
< gmaxwell>
Create a new wallet from a descriptor, or import a descriptor into a wallet.
< gmaxwell>
sipa: maybe you can work with achow 101 on a descriptor-lite implementation that would be forward compatible with the full thing later, and is enough to support the common hardware wallets?
< sipa>
gmaxwell: descriptors are also in the codebase
< sipa>
*already
< gmaxwell>
well I meant basically "enough stuff to be able to create an xpub with path autofilling watching wallet"
< sipa>
the hard part is integrating them in the wallet of course
< sipa>
but that isn't harder or easier depending on what the descriptors support
< achow101>
gmaxwell: the problem with that is the wallet format needs to change
< sipa>
i'll try to think it through
< wumpus>
any other topics?
< gmaxwell>
jonasschnelli: wanted to talk about encryption!
< wumpus>
#topic P2P encryption (jonasschnelli)
< jonasschnelli>
Not really... :) but happy to do
< gmaxwell>
heh, what just going to open that big PR and go on vacation? :P
< jonasschnelli>
The question for me now is, is the PR to large (2000lines) and how and when to add NewHope (quantum resistant handshake)
< jonasschnelli>
If it is to large, how to split it into PR that make partially sense on its own (could add the crypto stuff beforehand, but meh)
< gmaxwell>
so the PR can probably be broken into three parts: (1) prepatory refactors, (2) inclusion of new primitives, (3) the thing itself.
< jonasschnelli>
Yes. Agree.
< jonasschnelli>
But would we merge 2 without 3?
< jonasschnelli>
Maybe 2 with concrete chances that 3 gets merged
< gmaxwell>
Yes. Exactly.
< jonasschnelli>
1 will be very small
< gmaxwell>
Well if we changed our mind we could always take it out. We've put other primitives in in advance before.
< gmaxwell>
e.g. bip32 key derrivation went unused for years.
< sipa>
BIP32 derivation was in the codebase for 3 years before being used :)
< sipa>
jinx.
< jonasschnelli>
Indeed. But its 2018 now. :)
< jonasschnelli>
Okay. I'll do the split then.
< wumpus>
granted, that was before people neurotically deleted dead code :-)
< gmaxwell>
It's fine, I think that chacha stuff is mostly a distraction from the review of that PR. It needs to be reviewed but mostly for build integration, etc.
< gmaxwell>
wumpus: or even not-dead-yet code!
< jonasschnelli>
heh
< sipa>
wumpus: it wasn't dead! it had tests!
< wumpus>
gmaxwell: even preemptively
< cfields>
jonasschnelli: taking a quick look at your PR, was the handshake spec modified to be sent first thing? Or am I misreading the code?
< jonasschnelli>
Yes. I can also try to extend the existing ChaCha20 RNG to be used for the crypto part
< wumpus>
jonasschnelli: anyhow if it simplifies review, please do split it up
< gmaxwell>
jonasschnelli: when you split make sure all the parts are visible. E.g. I'll want to look at (3) while reviewing (1) just in case some refactor seems like it makes no sense on its own. :)
< cfields>
jonasschnelli: woohoo, thanks
< jonasschnelli>
gmaxwell: Yes. Good point. Will do.
< jonasschnelli>
I guess keeping the "overall" PR somewhere helps to see the goal.
< gmaxwell>
sipa has done that in the past, e.g. with the segwit rebase.
< jonasschnelli>
The unanswered question is when and how to add the NewHope handshake part
< wumpus>
yes, indeed, though we probably want to change the one in high-priority then
< jonasschnelli>
Yes. Lets remove it for now
< jonasschnelli>
(ill do that)
< wumpus>
ok
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Aug 23 20:00:12 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< gmaxwell>
jonasschnelli: well my thought is that we make another PR on top of these-- likely with two commits, one that just adds the code to the build, the other which crams it into the handshake, so we see the marginal cost.
< jonasschnelli>
Yes. Exactly.
< jonasschnelli>
gmaxwell: Do you know if it is sane to replace using SHAKE as the output "KDF/XOF" with HKDF?
< jonasschnelli>
Or would that be a custom version of NewHope?
< gmaxwell>
jonasschnelli: we could just take its output as is for now. Lets change the absolute minimum for the first pass.
< jonasschnelli>
okay
< cfields>
jonasschnelli: I'm lagging quite far behind, so "go read the full spec" is a fine answer if you'd like. But I don't see any extensibility in the handshake spec? What's the reason for hard-coding a bare 32byte key without some sort of versioning/typing?
< sipa>
cfields: versioning outside of the key is identifiable
< cfields>
got it.
< cfields>
thanks
< jonasschnelli>
yes..
< jonasschnelli>
cfields: also, the additional NewHope handshake is not in the specs right now.
< gmaxwell>
cfields: we are avoiding having any fixed bytes so that it's harder for dumb DPI to block the traffic. A new protocol can distinguish itself by being incompatible. :)
< jonasschnelli>
But IMO censorship resistance is not property of that proposal...
< cfields>
jonasschnelli: right, I was just trying to figure out how the transition would work. I understand the issue now.
< jonasschnelli>
Not having censorship resistance as a property of the new protocol, doesn't mean, we should use static, identifiable data if avoidable
< gmaxwell>
in general upgrading this stuff, even with versioning is messy.
< gmaxwell>
like you send key type B and then the other side doesn't support B only A.... so what, now you start over? all doable just fine.
< jonasschnelli>
I just figured out we don't keep node stats outside of the connection lifetime...
< jonasschnelli>
(except addrman's service bits)
< gmaxwell>
But the extra complexity makes me feel like we're not likely to do something like newhope until the next major protocol rework, if it doesn't go in the first ersion.
< jonasschnelli>
Yes. It should go into the first, deployed version.
< jonasschnelli>
But maybe not in the same PR
< jonasschnelli>
Leading to the possibility of incompatible master<->master peers, which I think is fine
< jonasschnelli>
(or merge them almost simultaneous)
< gmaxwell>
yea, incompatible master/master doesn't worry me too much. If we're worried we could add a hidden option to disable the crypto and default it to off until we're ready.
< wumpus>
putting it behind an option initially sounds like a good idea in any case
< MarcoFalke>
jonasschnelli: Appveyor is a different build system
< MarcoFalke>
You'd have to add new files to the cmake file
< MarcoFalke>
(Assuming that that is the issue)
< jonasschnelli>
So all new files need to be added to the Makefile and to a special CI file?
< wumpus>
we need documentation for this appveyor stuff
< jonasschnelli>
Sorry,... I missed the whole PR about that
< wumpus>
it's a different build system and no one really has an idea how it works
< gmaxwell>
what does it do?
< wumpus>
well maybe MarcoFalke does :)
< MarcoFalke>
no, lol
< achow101>
gmaxwell: it's for building windows things IIRC
< achow101>
building in a windows environment
< jonasschnelli>
Do we really need a Visual Studio CI?
< wumpus>
otherwise I'm going to ignore it
< wumpus>
and merge if only travis passes
< MarcoFalke>
I think it is fine to ignore for now because it is experimental
< achow101>
MarcoFalke: is there supposed to a button to restart appveyor builds? sf so, i don't see one
< wumpus>
jonasschnelli: I think it's hardly reasonable to expect people submitting PRs to update two build systems
< MarcoFalke>
achow101: hmm, yeah there should be a button
< wumpus>
especially if we all have no clue how it works
< achow101>
presumably ken2812221 knows since he was the one to suggest it?
< jonasschnelli>
Maybe it is possible to run the appveyor CI on PRs but not report back (with green-checkmark or red-cross)
< jonasschnelli>
I'm not going to touch an .vcxproj XML...
< wumpus>
ya it's a proprietary format too
< luke-jr>
smh
< ryanofsky>
would be nice if appveyor errors on github were only shown as warnings. it is actually pretty straightforward to edit .vcxproj files, though
< wumpus>
I'm sure it's learnable, but I think we already ask enough things from contributors
< wumpus>
not that it would hurt to have documentation on how to add files, even with the current build system
< MarcoFalke>
Hmm, anyone knows how to have one of the linux jobs in travis compile with clang?
< MarcoFalke>
I installed clang and then set CC=clang and CXX=clang++, but these are simply ignored
< MarcoFalke>
All those build systems are hokuspokus to me