< NicolasDorier>
however this seems complicated, and not possible today
< sipa>
there was a PR for adding a dump utxo set to the rest interface
< sipa>
but it required a non-released libevent or so
< NicolasDorier>
so instead I was thinking the following: What about Bitcoin Core share the txoutset hash on bitcoincore website or twitter or whatever (like every 1000 blocks). At BTCPay level, I would show the UTXOSet of their own node and check against the website
< NicolasDorier>
and display a warning if that differ
< NicolasDorier>
So the user could fast sync from untrusted source, and would get a warning if that differ from Bitcoin Core website/twitter or whatever publish mechanism
< sipa>
nack, we're not creating a blessed utxo set hash
< sipa>
way too much power for abuse
< gmaxwell>
ouch no, even suggesting stuff like that makes me feel really negative about the fact that the sotware even has a utxo hash at all.
< NicolasDorier>
Well this won't break anybody software. I could make this check only once as well
< NicolasDorier>
I see
< sipa>
NicolasDorier: just the mere concept of having soke sort of 'official' utxo set hash sounds unacceotablw to me
< sipa>
*some *unacceptable
< gmaxwell>
esp since we now have also seen what has happened with "check utxo" like validation in ethereum, -- it's become impratical to validate ethereum without just blindly trusting utxo hashes.
< NicolasDorier>
Yes I can understand. I am just worried that right now what we will see with those "Bitcoin Core in a box" is blind reliance on a pre-shipped UTXO Set
< NicolasDorier>
which is even worse than publishing your UTXO Set hash somewhere public and people could check it only once
< sipa>
i think we can probably at some point have a utxo set hash in the software, if it's accompanied with the usual review cycles, including automated CI that recomputes it... on a scale of twice a year
< NicolasDorier>
I think that would be acceptable
< gmaxwell>
NicolasDorier: if the "bitcoin core in a box" then the entire software could do arbritary things, the hash it gives is meaningless.
< sipa>
but then it's not just "whoever controls the website", but the same trust level as people already have in the sofrware anyway
< gmaxwell>
yes, an assume valid like thing would be fine, works like regular sotware review.
< gmaxwell>
and regular software integrity.
< gmaxwell>
(esp since its so trivial to review, it doesn't even increase review surface)
< NicolasDorier>
This would be perfect.
< NicolasDorier>
The point I try to solve right now is that with BTCPay I am shipping a docker-compose that everybody can run easily. All images can be independently reviewed and built by yourself if you want. This now works on raspberry pi, and I want to guide people on how to use it on raspberry. But the sync time makes it impossible, so I am searching a good enough solution better than just shipping an untrusted utxohash
< NicolasDorier>
such hash in the source code would be perfect
< NicolasDorier>
especially because this check has to be done only once
< NicolasDorier>
only when importing an outside utxoset
< NicolasDorier>
"Searching a good enough solution better than just shipping an untrusted utxo set" I mean
< NicolasDorier>
actually does not even has to be part of bitcoind process
< NicolasDorier>
it could just be a separate utility "bitcoinutxosetcheck utxoset.tar"
< NicolasDorier>
mmmh you know what. Actually I will just add in BTCPay a way to get the current UTXOSet hash, so someone who imported an untrusted UTXO set for his raspberry PI, can cross check with another of his own node fully synched server from beginning.
< NicolasDorier>
Or just cross check with a friend
< NicolasDorier>
that's good enough
< NicolasDorier>
and without controversy
< NicolasDorier>
Ok so I think I found the ideal UX.
< NicolasDorier>
I will add a feature in BTCPay to let people use an untrusted UTXO set.
< NicolasDorier>
BUT once the node is fully synched, I will show a warning popup which will never go away: "Please input the UTXO set hash of the current block".
< NicolasDorier>
I will not show the UTXO set hash of the untrusted node anywhere in BTCPay interface, so it will force them to search for it on a their own trusted node, or ask to a friend.
< NicolasDorier>
I think this is even better UX than embedding known utxo hash set anywhere into bitcoin core.
< gmaxwell>
I think that is severely worse.
< gmaxwell>
NicolasDorier: an integrated utxo 'assume valid' doesn't change the security model at all, -- if the software was malicious the user is screwed regardless.
< gmaxwell>
An "input one from somewhere" is almost effectively "hand blockchain.info(dejure) control over the network"-- the user then is screwed if the software is bad Or if some website is bad/hacked.
< NicolasDorier>
I want to protect against malicious untrusted utxo set, I assume the software is not malicious (as it can be compiled from sources by the user itself already)
< gmaxwell>
If you assume the software is not malicious then a hash in it is also not malicious.
< NicolasDorier>
argh
< gmaxwell>
Asking a website for the utxo hash is trusting that the website is not malicious-- might as well just have them process the payment.
< NicolasDorier>
Well your would need to compromise both the website utxo hash and the UTXO set untrusted archive
< NicolasDorier>
but yeah
< NicolasDorier>
shit :(
< kallewoof>
Would a peer message "getutxosethash" which returned a block height and utxo set hash be a long term solution to this or is there a better alternative?
< NicolasDorier>
it is time consuming and ddos vector so :(
< NicolasDorier>
it does not solve the underlying issue either
< aj>
if the trusted utxo set hash comes with the software, and is updated every ~6 months; then you'd download the software, get a utxo set hash that's say 2 to 8 months out of date, download the corresponding utxo set, verify it, then download 2-8 months of blocks since then, and you're set?
< NicolasDorier>
@aj given you have a trusted node, fully synched. And as a reviewer, you want to check if this UTXO Set Hash is correct. How would you do?
< NicolasDorier>
I could code this hash inside BTCPay. But how other reviewer can be sure that it is not malicious
< NicolasDorier>
(given they have their own node)
< aj>
NicolasDorier: invalidateblock the block after it, and run the rpc call to get the hash?
< NicolasDorier>
ah indeed that would work
< aj>
(which i assume is gettxoutsetinfo -> hash_serialized_2 ?
< NicolasDorier>
aj: Another idea: They use my hash to sync a new node. Then they compare the utxoset to their own node
< aj>
if they have their own node, why don't they just copy their utxo set across?
< NicolasDorier>
because it is pain in the ass. I want a user friendly way to put a big UTXO set tar somewhere on the cloud that people starting a new raspberry pi can rely on to get synched quick
< aj>
NicolasDorier: that sounds fine, seems odd to expect them to already have a node is all
< NicolasDorier>
aj: At least for the reviewers. People running those "bitcoin core in a box" are already blindly trusting binaries anyway.
< NicolasDorier>
but if somebody takes time to review the code, I want him at least to be able to check that the hash I hard coded is not malicious given he has his own indendant node.
< sipa>
NicolasDorier: i fear "user friendly" and "auditable full node security without central point of trust" are not really compatible
< aj>
NicolasDorier: err? i was thinking in-a-box users blindly trust the hash, but the software automatically checks the utxo set they download matches the hash; that way reviewers don't generally have to download your utxo set at all (they just check the hash matches the one their node generates)
< NicolasDorier>
aj: The problem is, if I ship the UTXO Set up to block 500.000 there is no way in Core that the hash of this UTXO set match their node. The only way is that they take my UTXO Set up to 500.000, then let it sync to the latest block, then check that the two nodes (the one they trust, and the one where they used this UTXO set) match.
< NicolasDorier>
"there is no way to *check in core" I mean
< sipa>
why do you need the utxo set?
< NicolasDorier>
sipa: well to check the payments are real? :p
< sipa>
NicolasDorier: why do you need the historical utxo set?
< NicolasDorier>
So that raspberry pi user can sync fast. They would download it the utxo set at block 500.000 then sync up to the latest block.
< sipa>
NicolasDorier: i feel that introducing a convenient way to run a "full nkde" that relies on too centralized ways of determining that set is far worse than using a loght client
< aj>
NicolasDorier: if you're doing rc1 of your next release now, why wouldn't you pick a recent hash, say 200 blocks old like 551600? invalidateblock/txoutsetinfo only takes a couple of minutes to run to review that?
< NicolasDorier>
aj: yep true, this would work
< aj>
$ time (blk=$(bitcoin-cli getblockhash 551601); echo $blk; bitcoin-cli invalidateblock $blk; bitcoin-cli gettxoutsetinfo | grep hash_serial; bitcoin-cli reconsiderblock $blk) # --> 666550387c3f5fa8e9353ce561fcee2a164022d31b97affd7c982388f19a4904as the hash, took about 1m12 all up
< NicolasDorier>
sipa: Given I trust my software and my software can check the tar shasum of the "historical utxoset at block 500.000", then it is fine.
< NicolasDorier>
Now how do I trust the software? By reviewing the code. But by reviewing the code I see this shasum of historical utxoset. How can I review this hash?
< NicolasDorier>
Either by doing what aj said (invalidateblock up to the height) or just using this UTXO set to sync up to now and cross check that UTXO set hash of this node is the same than another of my node that I fully synched from the beginning.
< NicolasDorier>
aj: though the invalidateblock is not possible if the node you trust is pruned
< aj>
NicolasDorier: going back a few hundred blocks should be fine even if it's pruned
< NicolasDorier>
Yes true, this would work as well
< NicolasDorier>
While I distribute the UTXO set centrally, and ship this hash in the product, anybody would be able to check against it.
< sipa>
i don't like "anyone can check"
< sipa>
people don't
< gmaxwell>
the ethereum ecosystem shows this pretty perfectly... node falls behind, blindly accept the state.
< gmaxwell>
people don't check. And any that did would just check some centeralized website. The security in that case is pure pretext.
< NicolasDorier>
Well as people blindly run binaries shipped by core, and don't check they can reproduce the build. But at least checking this hash is easier than doign a gitian build.
< gmaxwell>
a lot of people do check that they can reproduce the build, and publish attestations too.
< NicolasDorier>
then why would they not do it for the hash?
< aj>
sipa: the alternative to "don't check but can" that people will actually adopt is "don't check and can't" though, which seems worse...
< gmaxwell>
I think it's not worse.
< sipa>
aj: that's fair
< gmaxwell>
a _dishonest_ security model is worse than being insecure in practice and lying to the whole world and yourself about it.
< gmaxwell>
maybe more people should have given a shit since _2011_ that making it absurdly expensive to sync would be a problem.
< gmaxwell>
and instead worked to prevent that instead of taking an unsustainable load and doubling it.
< gmaxwell>
but whats past is past.
< NicolasDorier>
well this would have happened one day or another
< sipa>
NicolasDorier: just so we're talking about the same thing, i do not think it is inherently necessary that every full nkde forever validates all of history
< gmaxwell>
not necessarily, e.g. the work by phantomcircuit and luke-jr that showed the maximum rate of blockchain increase that would have kept sync time constant under the assumption of some year over year percentage increase in sync performance.
< gmaxwell>
in 2015 the figure for that would have been roughly 300kbyte blocks.
< gmaxwell>
in any case, water under the bridge.
< gmaxwell>
It sound like whats previously been proposed for utxo assumevalid would work fine for you, except it doesn't exist yet.
< sipa>
there are perfectly reasonable solutions to avoid that, including a deeply buried infrequently updated assumevalid utxo set hash
< NicolasDorier>
I would like at least that node which do not fully validate has a way to ring the bells if their utxo stop matching with other people, or with their own node on a different server which fully validated everything.
< gmaxwell>
Please do not poison the well by proposing "check against a website"... all that is going to do is create a big fight over it and cause people to also fight against more reasonable measures.
< NicolasDorier>
yeah no worry I won't do this
< gmaxwell>
if their software is malicious then there is no way they can count on it to warn them...
< aj>
gmaxwell: if their software isn't malicious but the utxo download site is, they could get a warning at least
< gmaxwell>
aj: download site? it should be getting it from the network and checking it against the software.
< aj>
gmaxwell: maybe, but there's a lot more implementation to do for that
< aj>
gmaxwell: well, unless you're happy with "network" being "torrent of the utxo set at a point in time"
< gmaxwell>
It's been implemented before.
< aj>
gmaxwell: for a utxo set up to 6 or 8 months old, or just the current (rolling) utxo set?
< gmaxwell>
I guess like actually improving the protocol might get in the way of five refactors a day.
< sipa>
come on
< gmaxwell>
aj: there was a snapshot one implemented previously.
< gmaxwell>
It has been _years_ now since we've made a protocol improvement, and when an need gets brought up for one it seems like it's just a question of how fast we could enable trusting a website...
< gmaxwell>
It's bad enough when people outside of regular development act helpless about limitations that could be improved.
< aj>
gmaxwell: url or search term suggestion for the previous implementation?
< gmaxwell>
some randos github repo-- they didn't hang around and keep trying to contribute when their efforts didn't make prompt progress, I'm searching
< cjd>
I was quite impressed by Peter's work on UTXO proofs
< sipa>
latest scaling bitcoin had a talk on improvements to cryptographic accumulators to make them more usable for this kind of application
< sipa>
yeah, it's in there; there is also a ml post
< sipa>
i have written a post on rolling utxo hashes (which don't permit compact proofs, but may help comparing utxo sets)
< gmaxwell>
aj: unfortunately the impl before came from someone who wasn't a regular so they didn't use our ordinary language to talk about it, so I'm having a damnd hard time grepping up the link.
< sipa>
gmaxwell: it was a bitcointalk post afaik
< cjd>
I'm kind of cold on anything that requires shor attackable crypto at the infrastructure level, but that said I think a lot of these things are still using hashes
< cjd>
And I distinguish between infrastructure level (hashes, merkle trees etc) from the spending level (signing a transaction)
< dongcarl>
A v0 WitnessProgram with a size that's not 20 nor 32 bytes is still a valid WitnessProgram, but will fail, is that a correct distinction?
< cjd>
it would be nice to go all in on the wild crypto and be able to make everything anonymous, but I think that should be kept as an option to put your money into the black box and then take it out again, knowing that some major advancement in math or quantum computers might cause the black box to implode
< gmaxwell>
sipa: ah, it was also several weeks of talk in IRC.
< gmaxwell>
which is what I was trying to find.
< sipa>
dongcarl: correct
< sipa>
dongcarl: though i'm not sure that that distinction is observable
< dongcarl>
sipa: `CScript::IsWitnessProgram` seems to not check that a v0 WitnessProgram is either 20 or 32bytes
< gmaxwell>
as there is no input that will hash to match it.
< sipa>
dongcarl: correct
< dongcarl>
right right
< gmaxwell>
dongcarl: a complicated thing with consensus rules is "which of the infinite set of implied consensus rules should be called consensus rules"?
< sipa>
but the only effect of something being a witness output is that it means a witness can be present when spending it
< sipa>
however, in the case of a v0 with not-20/32 size, there is no valid spend
< gmaxwell>
like, yes maybe bitcoin core has a branch that checks the size and terminates... but without that branch it would still reject those cases because the hash function (a size triggered OR of two functions) will never match for inputs of other sizes.
< sipa>
right, operational semantics can differ from the validaty rules...
< sipa>
as in one (invalid) case we may check signatures and in another invalid case we reject before reaching that point
< gmaxwell>
like it will also fail if your input is PUSH("RICK") OP_ROLL, but we wouldn't say that we have a consensus rule that the witness scripthash can't be PUSH("RICK") OP_ROLL.
< gmaxwell>
Even if we were to add code to specifically catch that case because some clown was spamming with it and we thought it would be useful to fail faster.
< gmaxwell>
(or more often, because being specific can make the semantics more clear)
< dongcarl>
gmaxwell: ah... makes sense... some rules are stated, and some rules are the consequence of other rules
< sipa>
dongcarl: in any case, by the terminology of the BIP and the source code, yes, a v0 21-byte output is a witness output
< gmaxwell>
and sometimes we make implied rules concrete ... and then only later take them out again.
< sipa>
however, whether it is or not has no effect (i think!) on any consensus observable effect
< gmaxwell>
as which of the implied consensus rules we implement concretely is just an implementation choice...
< gmaxwell>
sipa: that probably has some effect on RPC outputs.
< dongcarl>
sipa: gotcha, I'm looking at the test case PR for `IsWitnessProgram`, so wanted to double check
< dongcarl>
gmaxwell: Soo... consensus is really just implementation?
< sipa>
dongcarl: ultimately, yes - but i don't think that's the conclusion here
< sipa>
there is a set of intended rules, but there are still many ways of mapping those rules to code that are all identical in terms of observable behavior
< sipa>
and the implentation may choose to make distinctions that don't exist in the observable rules
< sipa>
for example, bitcoin core will use a very different codepath for an ecdsa signature that is too long vs one that is just invalid
< gmaxwell>
I normally think of the consensus rules as the MINIMAL set of rules that implement the 'consensus functionality'.
< sipa>
but the rules that people understand the system has doesn't make this distinction at all
< gmaxwell>
For implementation reasons we don't implement "the consensus rules" we implement some isomorphic set of rules. They behave the same for consensus purposes, but have some implementation advantages.
< gmaxwell>
E.g. cheap tests for expensive failure conditions... like checking the length of a witness hash, we don't have to-- a wrong length won't ever agree with the hasher, but perhaps the code is more obviously correct if the lengths are checked.
< cjd>
This all reminds me of the recent(ish) work that was done at inria on modelling the SSL state machine
< cjd>
Source of that handful of attacks which were published in the past 3 years
< dongcarl>
Ahhhh okay I think I get it... Sometimes short-circuiting (even when consensus doesn't dictate it) can have advantages in terms of implementation
< sipa>
dongcarl: exactly
< sipa>
consensus only outputs "valid" or "invalid"
< sipa>
it doesn't even report why something was invalid
< sipa>
(though that may in some cases be obserable on the P2P protocol by bans and corruptionpossible flag)
< gmaxwell>
though thats also one of the reasons I don't like having things like reject reasons visible... they're hard to make stable without really confining the specifics of the implementation.
< gmaxwell>
"Your thing was wrong for 5 reasons, but we're only going to tell you one of them."
< sipa>
and after a refactor, we'll tell you about another one instead
< Guest80>
Hello!
< Guest80>
Anybody here?
< gmaxwell>
nope
< Guest80>
Funny...
< Guest80>
;-)
< dongcarl>
gmaxwell: Huh... I was actually thinking that errors need error codes in Bitcoin... But perhaps that's a bad idea...
< Guest80>
Anybody here experienced with bitcoin core build?
< Guest80>
My simplified question is: Ho to build Windows installers?
< Guest80>
I can build Windows binaries but they all Stand Alone ones.
< Guest80>
Thanks, I'm familiar with the document you linked.
< dongcarl>
Guest80: you can build a Windows installer using other tools, it's not in the scope of the project I don't think
< Guest80>
I'm here because that document is no help at all.
< Guest80>
Thanks!
< sipa>
dongcarl: libconsensus will give you error codes
< sipa>
for script validity at least
< Guest80>
Bitcoin.org has Windows installers. They build them somehow.
< sipa>
Guest80: using a deterministic build environment
< sipa>
which is fully scripted
< sipa>
but runs in a linux vm
< dongcarl>
sipa: Oh cool... So how does it decide what error to point out?
< Guest80>
I build under Ubuntu 10.04.1 LTS
< Guest80>
That Ubuntu runs as a VM.
< gmaxwell>
dongcarl: well I don't like them, because either we don't keep them stable which maybe breaks things. ... or we have an implementation suicide pact where it's really hard to change stuff just due to preserving errors.
< dongcarl>
Or does it suffer from the problem of pointing out the first problem it finds and probably will point out a different one if we refactor?
< gmaxwell>
exactly
< sipa>
thankfully, that part of the consensus code (script interpreter) does not need much changes
< dongcarl>
Gotcha
< sipa>
the types of failures around blkcks and transactions changs more often
< promag>
sipa: GetAffectedKeys could be GetAffectedPubKeys?
< Guest80>
Again, the link you linked is irrelevant.
< Guest80>
I am building for Windows target under Ubuntu.
< Guest80>
I am not building under Windows.
< Guest80>
I can build "by the book". That's not the problem.
< sipa>
Guest80: yes, follow the gitian guide
< sipa>
promag: it returns key kds
< sipa>
key ids
< sipa>
key ids identify keypairs
< Guest80>
The problem is that the results of the build are Stand Alone Windows binaries, and not the Windows installer.
< sipa>
gitian also produces the installer
< Guest80>
And I need the Windows installer.
< Guest80>
Well, it does not produce the installer.
< sipa>
yes it does
< gmaxwell>
I believe the gitian build makes the installer. You need to get the extracted signatures for it to produce a signed installer.
< Guest80>
It collects everythin in one folder, but none of themis the installer.
< Guest80>
If the Gitanian build makes the installer than it does it with a special switch or something. NOT by default.
< instagibbs>
I will shut up about it now, with an issue open.
< instagibbs>
#14380 ready for merge?
< gribble>
https://github.com/bitcoin/bitcoin/issues/14380 | fix assert crash when specified change output spend size is unknown by instagibbs · Pull Request #14380 · bitcoin/bitcoin · GitHub
< bitcoin-git>
[bitcoin] vim88 opened pull request #14831: Scripts and tools: Use #!/usr/bin/env bash instead of #!/bin/bash. (master...proper_shebang) https://github.com/bitcoin/bitcoin/pull/14831
< bitcoin-git>
[bitcoin] ch4ot1c opened pull request #14832: docs: Add more Doxygen information to Developer Notes (master...improvement/doxygen-docs) https://github.com/bitcoin/bitcoin/pull/14832
< gribble>
https://github.com/bitcoin/bitcoin/issues/14380 | fix assert crash when specified change output spend size is unknown by instagibbs · Pull Request #14380 · bitcoin/bitcoin · GitHub
< BlueMatt>
#14196 also looks merge-able (its against 0.17 and the upstream forward-port was merged)
< gribble>
https://github.com/bitcoin/bitcoin/issues/14196 | [0.17][psbt] always drop the unnecessary utxo and convert non-witness utxo to witness when necessary by achow101 · Pull Request #14196 · bitcoin/bitcoin · GitHub
< dongcarl>
I see that kallewoof has posted a few clang static analyzer reports on the repo... Wondering if there's any effort to have a bot that does this automatically or have CI integration for this...
< kallewoof>
I thought practicalswift was our clang static analyzer bot