< gribble>
https://github.com/bitcoin/bitcoin/issues/14733 | P2P: Make peer timeout configurable, speed up very slow test and ensure correct code path tested. by zallarak · Pull Request #14733 · bitcoin/bitcoin · GitHub
< bitcoin-git>
[bitcoin] fanquake opened pull request #14853: depends: latest rapidcheck, enable property based tests on Travis (master...latest-rapidcheck) https://github.com/bitcoin/bitcoin/pull/14853
< fanquake>
Chris_Stewart_5 hopefully 14853 unblocks your progress in 14430
< fanquake>
Apparently travis can't open .zip files.. ?
< NicolasDorier>
gmaxwell, sipa, aj, we talked before yestersday about the "least worst" way to have UTXO Set snapshot and how it would be cool if, as part of the core release, a utxo snapshot hash could be included which contributors can sign against.
< NicolasDorier>
I tried to explain the downsides and documented the process about how people can verify those UTXO Set by themselves if they have another trusted node somewhere else.
< NicolasDorier>
I also documented how to sign against it.
< NicolasDorier>
I know how much you dislike the idea, but this still solve a problem and can be used responsibly. Please, if you have time, review this doc and let me know if I need to add additional warning or documentation. (or if you feel like to add your signature! :))
< hgfjkgh>
hello how are you
< gmaxwell>
NicolasDorier: This is how decenteralization dies, not with a bang but a wimper. Having a 300 block back snapshot effectively guarentees that any review process will be worthless.
< aj>
NicolasDorier: oh, i was thinking that was part of a "btc-in-a-box" releases, not core ones
< sipa>
NicolasDorier: if you need that frequent snapshots, i think you shouldn't use a full node; the security is an illusion, they're effectively trusting just you
< aj>
sipa: the idea wasn't to be that frequent, i thought -- you'd only do the snapshot every 6 months or so, it'd just be that "recent" to make it easy to calculate
< gmaxwell>
300 block snapshots effectively means that the node is not fast enough to catch up after a bit of an outage.
< sipa>
aj: yes, that would be much more reasonable, and be compatible with a public audit
< gmaxwell>
which is what you have in ethereum,-- if you fall behind you have to re-quick-sync.
< provoostenator>
I'd much rather see something along the lines of #9483 but using neutrino filters, and then catching up with full blocks in the background over time.
< gmaxwell>
provoostenator: there is no reason to use neutrino filters for the sorts of use case contemplated here, AFAIK.
< provoostenator>
The idea is spin up the node as quickly as possible with as little trust as possible.
< gmaxwell>
provoostenator: in this application you are processing payments. There will be no payments to you before to started in the first place, so no need to go far back in history.
< luke-jr>
is it time to make a wiki page reflecting devs' opinions on block size/weight reduction softforks? ;p
< albypaul>
is there a way to change the difficulty of bitcoin core if you are mining off you're own computer just a question i need to ask.
< luke-jr>
albypaul: #bitcoin
< sipa>
albypaul: use regtest mode
< gmaxwell>
provoostenator: and so if you start near the current tip, you just need to use an average of 14kbit/sec to watch for your payments, no filters required.
< provoostenator>
Right, without history you can just check whats in a block and trust proof-of-work for new payments, until you've verify all history.
< luke-jr>
(unless you really mean testing purposes)
< gmaxwell>
provoostenator: yup.
< albypaul>
none testing
< luke-jr>
albypaul: if you're not testing, then ask in #bitcoin to learn why you can't
< albypaul>
thank you luke
< provoostenator>
So I wonder what the first step is for anyone who wants to tackle this (sync headers to tip, accept new blocks based on PoW check, then sync blocks retroactively).
< gmaxwell>
provoostenator: jonasschnelli previously implemented much of it
< provoostenator>
Yes, and that PR is in limbo, partially because it was too big and needed splitting.
< aj>
luke-jr: i think more info about effects of block weight limits would be good. i guess you're thinking of the argument along the lines that 17% pa increase in network speed means 400kB/block if the 250GB of block data is the limiting factor?
< luke-jr>
aj: the current blockchain size is too big, so we probably need to stick to 300k as ~17% of the last reasonable blockchain size
< luke-jr>
aj: but an opinions page could present multiple opinions ofc
< luke-jr>
(reasonable meaning, when we had a safe level of users running full nodes of their own)
< gmaxwell>
luke-jr: I don't think thats a realistic answer. besides, 2days increase is already too much... and 17%pa doesn't really reflect the available hardware.
< gmaxwell>
I don't believe e.g. the things cited for btcpay (cheap vpses, rpis) are speeding up at 17%pa.
< aj>
luke-jr: i'd find the reasoning more interesting; like what's "too big" mean? initial sync time for sure, but is there a point when the utxo set is "too big" too (i mean, it's only 50M atm, which doesn't come close to a utxo for every person and company eg), etc?
< gmaxwell>
We've had pretty extensive descriptions of things we can do to fast start, from SPV start so you can get paid immediately with spv security, to UTXOassumevalid. We just don't have enough people right now who feel qualified to move the protocol ball forward.
< gmaxwell>
aj: 50M!?! 2500M you mean...
< gmaxwell>
oh maybe 50 million entries.
< aj>
yeah, entries, not bytes
< luke-jr>
gmaxwell: what do you mean? we need to go lower? :|
< provoostenator>
gmaxmell it sounds like SPV start has the biggest bang for developer & review buck
< provoostenator>
*w
< luke-jr>
SPV start isn't really a solution to an indefinitely growing IBD time
< luke-jr>
although maybe it would make the current chain size more tolerable to users
< gmaxwell>
luke-jr: thing is that a "x day sync time" is a total non-issue for many users so long as they can send and recieve payments right away.
< gmaxwell>
so it would immediately take things from unusable to "okay" for a large class of users.
< luke-jr>
gmaxwell: and can use Netflix in the meantime :/
< gmaxwell>
luke-jr: yes, esp if its in the background we could reasonably rate limit the sync.
< provoostenator>
Once people are hooked, they'll accept slower sync :-)
< luke-jr>
gmaxwell: but if 17% is too much, going based on the current blockchain size might just put us back at 300k anyway :<
< gmaxwell>
I think it just isn't so simple. 17% is perhaps too little for top end deployment, but it's probably too much for bottom end deployment.
< gmaxwell>
the fact is that the competition is "use paypal" or "use bitpay", and those don't take much resources and don't grow at 17% per year.
< luke-jr>
people choosing paypal doesn't harm Bitcoin's security, though
< gmaxwell>
And so low end SBC hardware or VPSes will continue to target low end usage, which will work with centeralized bitcoin alternatives.
< gmaxwell>
luke-jr: well it does when to compete with paypal others make centeralized bitcoin solutions.
< luke-jr>
I must be misunderstanding you somehow.
< provoostenator>
I took a stap at rebasing something that I thought was a prerequisite for SPV sync #13937, but a) this was not easy, and b) I'm still not sure if it really is a prerequisite.
< luke-jr>
provoostenator: it's probably one possible approach to the end goal
< luke-jr>
with some useful side effects
< provoostenator>
So I'm trying to figure what to tell anyone who wants to work on this problem, what would be something manageable.
< gmaxwell>
luke-jr: even if you don't care about compeating with paypal, other people in the bitcoin ecosystem do. And they'll make tradeoffs as required to do so, and drag the ecosystem in that direction, especially if people who don't so much care about paypal ignore the requirements.
< gmaxwell>
provoostenator: I can't give you advice right now because I've lost track of that work, but I'll make an effort in the next day to catch up on it.
< provoostenator>
It touches a lot of very critical code which just takes time to understand, so it may simply be something that can only be done with lots and lots of time and dedication.
< luke-jr>
gmaxwell: ah, like moving economic activity to centralised off-chain solutions?
< provoostenator>
gmaxwell: thanks, no rush at all
< gmaxwell>
luke-jr: worse, really bad ones, rather than like.. more thoughtfully created ones.
< gmaxwell>
and then also advertise them as decenteralized. :P
< luke-jr>
well, if they move to altcoins, that's not so much Bitcoin's problem..
< luke-jr>
hence why I'd like to see a certain altcoin survive some forking mess
< sipa>
luke-jr: if chasing people away to alternatives is a solution, then 0 kb block size limit is ideal.
< gmaxwell>
:P
< gmaxwell>
in any case, I feel like this is a tangent, my point was that it pays to be considerate of a wider class of usecases than you would naturally care about, because we also want the participation of those people.
< luke-jr>
sipa: well, chasing away centralising people would only be a means to a goal (keeping Bitcoin itself secure)
< gmaxwell>
A luke-jr only currency would not have any scaling problems, at all... and that would be a problem for it. :P
< luke-jr>
0k block size kinda throws the baby out with the bathwater
< sipa>
luke-jr: there are no "centralized" and "decentralized" people; there are people who care more about decentralization than others, but very few have it as an absolute requirement
< gmaxwell>
luke-jr: it's not really that anyone (well not manyone) wants a centeralized solution. They want something like cheap instant payments now, and centeralized is perhaps just how they know how to get it.
< sipa>
if the only thing that works is centralized, it will be what most, if not everyone, uses
< luke-jr>
gmaxwell: hopefully Lightning helps with that
< gmaxwell>
Absolutely.
< provoostenator>
And there's people who care about decentralalization just as much but don't fully appreciate the risks, especially the more subtle problems.
< gmaxwell>
But thats also just one thing in a broader spectrum of improvements.
< gmaxwell>
And NicolasDorier's use case is _tremendously_ important. I feel like we've had this backwards. btcpay was needed in 2011, desperately needed in 2012... and for lack of it, the supporting infrastructure has not become strong in the ways that btcpay needs it to be strong.
< gmaxwell>
instead, most parties were using a hosted centeralized solution.
< luke-jr>
eh, I don't think smaller blocks is at odds with BTCPay - it sounds like it would even help
< luke-jr>
especially considering BitPay apparently doing stupid things with extra sweep transactions
< provoostenator>
I think gmaxwell meant things like SPV sync with "supporting infrastructure"
< gmaxwell>
It would have, sure. Though it's not the point I was trying to make I think that the whole blocksize discussion would have gone very differently if the ecosystem was using btcpay and not bitpay a few years ago.
< gmaxwell>
provoostenator: right.
< gmaxwell>
And faster sync, and UTXO assume valid, and better backup, and god knows whatever problems we'll encounter if people really start processing their own payments under live fire at scale.
< gmaxwell>
(and, perhaps, lightning)
< luke-jr>
I don't think these approaches needs to be exclusionary to others (like smaller blocks)
< luke-jr>
blockchain size just happens to be the one irreversible factor, so IMO somewhat of a priority
< gmaxwell>
and not only did that harm users (though it seems not too much, save for a few merchants that got blacklisted) and some dumb political consequences like bitpay having way too much influence for their level of sanity... it has resulted in weaker ecosystem.
< gmaxwell>
luke-jr: it's not an irreversable factor. At least in terms of initial time, I think it's substantially addressed by UTXOassumevalid.
< provoostenator>
It also created a vicious circle where a shitty user experience means merchants don't see a lot of BTC revenue, so they're not inclined to look for competitors.
< gmaxwell>
Because then you have to ask what growth rate allows keeping the last year of blocks at the same sync time, and the answer is much much larger.
< provoostenator>
That will fix itself though if the deplatforming continuous and the alternatives get increasingly better.
< luke-jr>
gmaxwell: UTXOassumevalid is no longer a full node security level if it means what it sounds like
< gmaxwell>
luke-jr: I believe it has the same security model that users get right now.
< sipa>
luke-jr: if it isn't, then running a full node you haven't implemented yourself isn't either
< luke-jr>
provoostenator: my dedicated server provider accepts bitcoin via BitPay, and I just use PayPal nowadays instead because BitPay screwed up one too many times
< luke-jr>
sipa: you *can* audit full node code. If people rely on UTXOassumevalid, they *can't* audit the hash
< sipa>
sure they can; run a node with utxoassumevalid disabled
< provoostenator>
luke-jr: my dedicated pizza provider accepts bitcoin via BitPay, and I generally finish my pizza before it's confirmed, depsite all the error messages and RBF fud.
< luke-jr>
sipa: then they're not relying on it
< gmaxwell>
luke-jr: basically the user's software knows about a utxo state as of some height 6-12 months ago. And if the best chain is that chain, they'll skip validating and assume it valid. Now, it _could_ be invalid BUT the software could have a subtle one character backdoor burried it it too-- we endeavor to use public review to prevent both cases.
< sipa>
luke-jr: i believe more people will recalculate the utxo hash than there are people capable of reviewing the source code
< sipa>
(given sufficiently accessible processes around it etc)
< gmaxwell>
luke-jr: you audit it be running without, (e.g. on a faster system) or using an old node you had before. Or trusting any of many other people who do review it to catch issues and sound an alarm you'll hear, same as with the software in general.
< gmaxwell>
sipa: that appears to be the case for the current assumevalid vs changes to crypto code. The latter almost no one review, and AV pull requests get a bunch of people checking them.
< provoostenator>
So this UTXOassumevalid would use a much older block than we do with assumevalid?
< sipa>
rolling utxo hashes would make this much simpler, as you could at basically no cost let every node its utxo hash at every block and log it
< gmaxwell>
provoostenator: somewhat older because of a pratical issue of managing the snapshots.
< gmaxwell>
sipa: which is why that idea was created. :)
< gmaxwell>
(at least as far as I'm concerned)
< sipa>
gmaxwell: i know.
< gmaxwell>
provoostenator: basically if you have an UAV of height X then you need to sync from peers with that snapshot or otherwise full sync... and it takes space to store them, so they can only be so frequent.
< provoostenator>
You're putting the full snap shot in the binary, right? Not a hash and relying on P2P.
< gmaxwell>
provoostenator: it's reasonable to expect nodes to keep the last two or something, but not dozens of them.
< sipa>
no, relay over p2p
< gmaxwell>
provoostenator: over P2P, otherwise its too hard to distribute the software without extensive centeralization.
< sipa>
if we can relay blocks over P2P, we can definitely do the same with a utxo set (which is smaller than the blockchain!)
< gmaxwell>
And ideally we'd split up the data so that the peers you fetch it from don't have to have the whole thing on any single peer.
< provoostenator>
Right, that's gigabytes. Would be easier with the accumulator.
< sipa>
provoostenator: well in a hypothetical future where the utxo set is replaced with an accumulator, and txn contain inclusion proofs... there is no need for a utxo set anymore
< gmaxwell>
so for example, we could have each peer keep 1/8th of the last two snapshots, so even ignoring compression, the overhead is 1/4 the size of the utxo set. Any release just refers to the most recent one.
< sipa>
you'd just download the accumulator (which based on the technology could be a few megabytes, a few kilobytes, or even just a few bytes)
< provoostenator>
sipa: but you could still still use an assume valid accumulator state in the same vain
< gmaxwell>
unfortunately basically all those schemes shoot bandwidth or CPU through the roof, which it's not like we have either of those in surpluse. :)
< bitcoin-git>
bitcoin/master 5ab5341 Wladimir J. van der Laan: Merge #14841: consensus: Move CheckBlock() call to critical section...
< gmaxwell>
provoostenator: yes, that just saves the 2.5gb or whatever utxo set download. Which would be nice.. but we're orders of magnitude away from that being our biggest concern now.
< provoostenator>
^ quite possibly the scariest commit name one can thing of
< provoostenator>
It has the words consensus, check (block) and critical.
< gmaxwell>
fortunately it just moves a lock.
< wumpus>
that's good, maybe it will prompt some people to pay attention :)
< sipa>
ha
< gmaxwell>
not riskless, at all, no code change really is.
< gmaxwell>
I feel a lot more confortable about that change than the unpatched code! (who needs locks!)
< provoostenator>
Speaking of baby steps to SPV sync: #13930 documentation change just got blessed by bluematt (who wrote the original comment that I based this on)
< gribble>
https://github.com/bitcoin/bitcoin/issues/13930 | Better explain GetAncestor check for m_failed_blocks in AcceptBlockHeader by Sjors · Pull Request #13930 · bitcoin/bitcoin · GitHub
< gmaxwell>
[back to UTXOAV] we even have some nice designs that get you a cool property that with an 8way split you can sync off _any_ 8 peers, they don't need to be particular peers that have the parts you want.... so no issues with some attacker DOS attacking one slice to prevent sync.
< provoostenator>
Thanks sounds pretty cool, looking forward to read about it.
< gmaxwell>
well, you might have to build it if you want to read about it. :)
< bitcoin-git>
bitcoin/master fe1ff50 Chun Kuan Lee: Hide spendable label if priveate key is disabled
< bitcoin-git>
bitcoin/master 82d6c5a Chun Kuan Lee: gui: Show watch-only eye instead of HD disabled
< bitcoin-git>
bitcoin/master ed12fd8 Wladimir J. van der Laan: Merge #13966: gui: When private key is disabled, only show watch-only balance...
< bitcoin-git>
[bitcoin] laanwj closed pull request #13966: gui: When private key is disabled, only show watch-only balance (master...ui-disable-privkey) https://github.com/bitcoin/bitcoin/pull/13966
< NicolasDorier>
gmaxwell: Unsure I understand "This is how decenteralization dies, not with a bang but a wimper. Having a 300 block back snapshot effectively guarentees that any review process will be worthless.". I think I will do such snapshot only once every 6 months. It is only important for new nodes. And people can verify by themselves if they have their another full node they trust somewhere else.
< NicolasDorier>
I tried to explain the dangers in the README, let me know if I missed something
< Chris_Stewart_5>
Is fanquake on github also fanquake in here? I forget. To reproduce failures on builds with different envs is the path of least resistant just using some sort of VM? Is there docs on that in our github?
< bitcoin-git>
bitcoin/0.17 fcdea8a Andrew Chow: Drop the unnecessary UTXO based on the UTXOs present, not on earlier wallet things...
< bitcoin-git>
bitcoin/0.17 fcefc68 Andrew Chow: Convert non-witness UTXOs to witness if witness sig created...
< bitcoin-git>
bitcoin/0.17 3362a95 MarcoFalke: Merge #14196: [0.17][psbt] always drop the unnecessary utxo and convert non-witness utxo to witness when necessary...
< sipa>
NicolasDorier: it seems people misunderstood your document then
< sipa>
NicolasDorier: but i'm still worried about a lack of review procedure... what prevents you from signing a snapshot every day?
< sipa>
the more frequently you do it, the better the user experience will be
< harding>
NicolasDorier: you might get a clearer and less controversial security model by using SPV for BTCPay and having it connect to a single trusted node. That way people can run BTCPay on low-power hardware and the people who want to do personal full verification can connect to their own node from BTCPay. The people who don't care about doing full verification can just connect to someone else's node to use BTCPay, and you can have a
< harding>
warning on one of BTCPay's pages telling them that they're trusting the operator of the node at 1.2.3.4 to choose the best chain for them.
< harding>
NicolasDorier: oh, another nice advantage of that approach would be on moderately-powerful hardware you could start in trust-third-party SPV mode for almost-immediate usability of BTCPay. At the same time, you could start a local full node and let it start syncing. When the local node has synced after a day, a week, or even a month, BTCPay can seemlessly switch over to using that to eliminate trust on outside nodes.
< phantomcircuit>
interesting, i have a vm where select() calls are regularly being interrupted by signals on master
< phantomcircuit>
hmm let me double chec that isn't master actually
< phantomcircuit>
indeed it is
< esotericnonsense>
if you do have a utxo snapshot, the idea should be that the period is sufficiently long that an alarm can be sounded if the snapshot isn't actually correct.
< esotericnonsense>
i'm not sure if this is just an assumed obvious point in the previous discussion (i think that's what gmax is trying to get at with the review period).
< esotericnonsense>
if you have something super short then you have to have automated "alarms" and some way of disseminating very very quickly 'oh hey the snapshot in software XYZ is nonsense'.
< esotericnonsense>
it's not about 'decentralization' vs 'centralization' at all. a utxo snapshot in some repository somewhere is a 100% centralized source of truth regardless of whether it is the genesis block or 1 block ago. but if it's very very long ago then people have time to talk about it and warn users away from using bad software.
< gmaxwell>
esotericnonsense: and if your have automated alarms then you have to worry about the automation being compromised.
< esotericnonsense>
yes. e.g. externally someone can eclipse attack the nodes used for the checks.
< esotericnonsense>
sorry, I think I wasn't clear enough, a short period is useless and shouldn't ever be used.
< esotericnonsense>
the costs outweigh the benefits in most cases I would think for anything shorter than _at least_ a few weeks in the current ecosystem but probably months or morw.
< esotericnonsense>
more*
< gmaxwell>
Another way I look at short tims is this: Say given your hardware and sync time tolerance, you want a snapshot no less than X blocks back. This also means that if you're down for more than X blocks, you'll want to snapshot instead of catch up, becuase your tolerance for downtime is not going to be better than your tolerance for initial sync (and probably a lot worse).
< gmaxwell>
this is the factor that has resulted in the norm in ethereum that if your node has fallen behind, you resync it. ... maybe it fell behind because it was too slow, because your link was down, or because miners decided to break the rules. ... in all cases, "fixed by resync".
< gmaxwell>
I'm told that at least one exchange has their ethereum nodes do that automatically if they fall behind some block explorer they poll.