< gmaxwell>
promag: maybe everywhere we should split min_conf into min_conf_trusted and min_conf_untrusted. Most wallet behaior defaults to 0,1.
< gmaxwell>
(or more specifically, the coinselection code by default does something like, 6,6 and if that fails 1,1 and if that fails 0,1.)
< phantomcircuit>
gmaxwell, what?
< phantomcircuit>
of min_conf for our own transactions nvm
< droark>
Hi. I have a question. On mainnet, about how long does Core need to run before the fee estimation code is considered accurate?
< sipa>
about twice as long as the delay you want to estimate for
< droark>
Thank you.
< droark>
Also, I seem to recall there being some reason why the code can't provide an estimate for TX insertion within one block. I can't remember it, though. Does it have to do with statistical variance?
< sipa>
yes, the next block may be 1 second away
< sipa>
no fee can give you a 95% chance of being included in that
< droark>
Thanks. One last question for now. To clarify, when you say twice as long as the delay, do you mean actual time or # of blocks? I assume the latter but it sounds like I might be wrong.
< sipa>
#blocks
< droark>
Thanks.
< wumpus>
unknown options in the config file are ignored, unknown command line options give an error, I think that's a good middle ground
< wumpus>
if you explicitly provide -wallet= on *the command line* with a -disablewallet build I think it's fair to error out, you're requesting something it cannot do
< wumpus>
while if there's still left-over wallet configuration in the configuration file, well that probably shouldn't trip up too bad
< promag>
wumpus: I tend to agree with that
< fanquake>
wumpus I think #14741 and #14319 can go in
< fanquake>
wumpus yep. I was going to add that as my "high priority" PR. Once that's in I'll backport to 0.17, then start looking at 5.12
< fanquake>
Guess I'll add something else to HP
< wumpus>
heh! it's good to have this in, I think, so that it (hopefully) gets some manual testing before release, qt relies mostly on manual testing afterall
< fanquake>
hopefully nobody has been trying to use the (now disabled) lcd number functionality :p
< wumpus>
hah just looked what that does and it really emulates a LCD display on the screen, no I don't think we plan on using that :p
< wumpus>
always good to speed up qt compile by removing unnecessary modules
< wumpus>
I'm happy how fast we got the boost compile already in the same way
< fanquake>
Yes. A NO_QT=1 depends build is quite fast
< wumpus>
yep even on slow hw
< wumpus>
speaking of which, my RISC-V node still runs fine <3
< fanquake>
nice
< fanquake>
wumpus have you found a RISC-V board that will run Qt yet :o
< wumpus>
fanquake: I'm fairly sure it would run qt (using remote X server) :-) but no, I don't have the extension board with PCI-E functionality so I can't actually connect a monitor
< wumpus>
SiFive Unleashed + Extension board + AMD gfx card could run qt, but the former two are really hard to get hold of. I think that's going to improve though, more RISC-V cores are in the works, maybe next year...
< fanquake>
wumpus cool. Will have to track something down to experiment with.
< wumpus>
I really hope there will be more affordable boards that can run Linux, next
< luke-jr>
would be nice to get POWER gitian builds merged, considering it's a much better deal than RISC-V etc
< luke-jr>
shesek: please fix your connection problems sometime in the next 4 hours
< wumpus>
luke-jr: what's blocking that?
< luke-jr>
wumpus: no idea; there's a minor rebase needed, but it sat idle for some time before that
< wumpus>
that's a *weird* issue, why would that only happen on one platform
< wumpus>
could be a compiler bug
< luke-jr>
yes, it's not very clear if the compiler or Linux is screwing up here
< luke-jr>
since the VDSO is provided at runtime, I would expect Linux
< luke-jr>
but this VDSO code hasn't changed since 2010
< cjd>
Could have been interesting to see the backtrace on the gettimeofday() to see if there is perhaps something weird about where the timeval is placed
< cjd>
I could imagine something like timeval placed on a 32 bit boundry, linux thinks it's supposed to be on the 64, trashes the stack cookie
< cjd>
but I only looked at it for 5 min so my 2c is probably overpriced
< wumpus>
looks like travis is failing on all new PRs
< gmaxwell>
wumpus: looking at src/test/validation_block_tests.cpp it's totally misusing the rng in the test.
< sipa>
why did travis not detect this in the pr?
< gmaxwell>
I don't even really get why this paticular test exists-- like it's just starting lots of threads that call ProcessNewBlock at once. But ProcessNewBlock pretty much instantly takes a lock.
< wumpus>
yeh it's clearly sharing a FastRandomContext between multiple threads
< sipa>
actually, it's good to know tsan catches this
< wumpus>
sipa: hmm, maybe because your PR was from before TSAN checking was added
< sipa>
ah
< gmaxwell>
if so, how come it wasn't caught when tsan checking was added?
< gmaxwell>
simple fix, and just a test bug regardless.
< wumpus>
it doesn't re-run all the travis runs for allPRs on every merge
< gmaxwell>
ah.
< wumpus>
might be an old cache, or who knows
< gmaxwell>
though I still don't really get the utility of that test.
< wumpus>
it's pretty nice that this was aught and that you were able to decipher the error :)
< gmaxwell>
for some reason the tsan output is a little weird, it's only giving one side's backtrace.
< gmaxwell>
Not my first time decyphering tsan output. :P
< provoostenator>
Meeting?
< wumpus>
I guess it tests the locking in ProcessNewBlock in case that becomes less granular inthe future
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu Dec 13 19:00:20 2018 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< wumpus>
the poll one should be (almost?) ready to merge
< provoostenator>
Nominating #11082
< gribble>
https://github.com/bitcoin/bitcoin/issues/11082 | Add new bitcoin_rw.conf file that is used for settings modified by this software itself by luke-jr · Pull Request #11082 · bitcoin/bitcoin · GitHub
< provoostenator>
Needs rebase, but really could use more review before that.
< wumpus>
provoostenator: already needs rebase now
< wumpus>
(and has, for 24 days)
< jonasschnelli>
provoostenator: Maybe take that over from luke-jr
< sipa>
does it need concept discussion?
< jonasschnelli>
probably...
< provoostenator>
I will eventually, but I have 15 other thing on my list. I also think it needs concept discussion.
< luke-jr>
it doesn't need taking over, although if someone wants to rebase it sooner, I can push that
< provoostenator>
Though we've had IRC chats about it before and everyone seemed to think it was at least a reasonable approach, compared to alterantives.
< wumpus>
if it needs concept discussion, it definitely doesn't belong on the high priority for review list
< wumpus>
yes, I think it's a reasonable approach
< provoostenator>
Alright, maybe add it after rebase?
< wumpus>
I dread making init option parsing even more involved, but it's better than the alternatives
< provoostenator>
Dynamic wallet loading also needs this to make it good UX.
< sipa>
yeah, fair
< wumpus>
this needs *very* good tests
< provoostenator>
Otherwise you'd have to manually load each time you start.
< sipa>
provoostenator: i was expecting that to go in the qt settings
< wumpus>
especially for the qt case ast hat's even crazier
< wumpus>
how many levels of overlapping options can you have :)
< provoostenator>
It has a bunch of Boost tests, but can probably use some QT tests and functional tests.
< wumpus>
though this is supposed to get rid of most of the qt settings I guess?
< gmaxwell>
obviously we should store what wallets get loaded in wallet.dat...
< jonasschnelli>
Overlapping options is already problematic on the QT layer...
< provoostenator>
Well, fewer levels thanks to my followup #12833
< provoostenator>
But I'm reluctant to add more settings, as that makes rebasing 12833 a pain.
< wumpus>
luke-jr: so eventually it'll replace non-pure-user-interface settings in the qt settings, I mean?
< provoostenator>
(and that one also needs tests, I haven't delved into how to write QT tests yet, can someone make me a Python framework?)
< luke-jr>
wumpus: ideally
< provoostenator>
Yeah the idea is to get rid of QTsettings, with the exception of the datadir.
< jonasschnelli>
I guess the QT settings files will always be there for window sizes and such... but non critical settings for sure
< wumpus>
qtsettings is fine for *ui* settings
< wumpus>
such as the last position of the window
< provoostenator>
And what jonasschnelli says.
< wumpus>
and things like that
< jonasschnelli>
indeed
< wumpus>
but not for settings shared with the core
< provoostenator>
But not stuff that's shared with bitcoind like prune=
< jonasschnelli>
yes
< wumpus>
such as dbcache, etc
< wumpus>
right.
< jonasschnelli>
That was just a bypass of not having a way to write to a config section
< luke-jr>
?
< jonasschnelli>
We wanted dbcache, proxys in Qt but didn't had a way to write to the config, so we added another layer
< wumpus>
yep
< jonasschnelli>
Which is no longer a clever approach once we have rw_config
< sipa>
i'd be more comfortable if the set of modifiable settings (the list of wallets?) were distinct from those that aren't
< wumpus>
we definitely don't want to write to bitcoin.conf directly, but having a separate configuration file that's writable is fine
< sipa>
and have the GUI (where you really want everything to be modifable) directly modify bitcoin.conf
< wumpus>
noooooo
< wumpus>
don't write to bitcoin.conf, never
< sipa>
if you use the GUI, the GUI manages the settings
< wumpus>
conf should be read only
< sipa>
if you don't, the config file does
< wumpus>
we've been over this, please
< sipa>
right, or have a separate config entirely
< wumpus>
let's not change the idea now
< sipa>
i really dislike having a UI edit things at the same level as the user is supposed to
< sipa>
okay.
< * sipa>
shuts up
< wumpus>
we can have this discussion for a few years then never decide to do anything
< provoostenator>
There's also a bunch of settings that can go straight into the wallet payload, like RBF and address type (though not their defaults).
< jonasschnelli>
maybe this also raises the question how to distinct global settings between wallets
< provoostenator>
That also simplifies QT, because wallet settings can be updated without any of the QTSettings stuff.
< wumpus>
I'm unconfortable with writing to bitcoin.conf directly, as it is specified with -conf, which might point to a read-only configuration file, it should not be assumed it is writable
< wumpus>
putting settings in the wallet again?
< gmaxwell>
ugh
< sipa>
ugh
< wumpus>
yea deja vu...
< gmaxwell>
putting settings in the wallet I think is even worse now that we have multiwallet.
< jonasschnelli>
wumpus: there are the wallet flags now...
< provoostenator>
wumpus: yes, that was the idea. Wallets have meta data entries. We already put binary flags in the wallet.
< jonasschnelli>
but yeah...not settings
< sipa>
provoostenator: originally all settings were in the wallet, it was pretty bad :)
< luke-jr>
it might make sense for some subset of settings
< wumpus>
sure, wallet-specific metadata should be stored in the wallet
< wumpus>
but not settings
< gmaxwell>
so you load another wallet and then your software starts behaving differently, madness!
< provoostenator>
Yes, obviously only wallet settings.
< jamesob>
write to wallet.conf.qt in datadir, have that override -conf settings?
< gmaxwell>
yea sure wallet specific metadata sure fine whatever.
< jamesob>
*bitcoin.conf.qt
< wumpus>
jamesob: that's exactly the idea behind the bitcoin_rw.conf
< jonasschnelli>
I just though about address type per wallet,... how do we handle different address types with different wallets?
< luke-jr>
eg, you might have a non-segwit wallet, and a segwit wallet
< provoostenator>
See the list in the issue I linked to above.
< jonasschnelli>
*thought
< jamesob>
wumpus: my bad, getting to the meetin glate
< wumpus>
jamesob: it contains writable settings which override what is in bitcoin.conf
< wumpus>
jamesob: this means being able to edit settings through RPC as well as the GUI
< jamesob>
oh cool
< provoostenator>
Some of these make sense int he wallet: addresstype, changetype, discardfee, fallbackfee, keypool, mintxfee, paytxfee, txconfirmtarget, etc. For others it doesn't make sense.
< gmaxwell>
I don't see how fee settings make much sense in a wallet.
< wumpus>
I'm not sure
< gmaxwell>
Addresstype I could buy. maybe.
< provoostenator>
gmaxwell: to make them behave consistently.
< jonasschnelli>
Addresstype certenly,... fees, probably not
< sipa>
addresstype will go away in the brave new world future anyway
< wumpus>
yes
< gmaxwell>
keypool no, it almost could go away.
< jonasschnelli>
But I know a lot of people running a SW and a non-SW wallet in parallel
< provoostenator>
But maybe fee preferences are more mempool weather dependent than wallet specific.
< gmaxwell>
jonasschnelli: sounds brain damaged.
< gmaxwell>
jonasschnelli: do you mean they just want to get keys with different address types?
< jonasschnelli>
It may be,... but it's just how transition phases happens
< provoostenator>
"keypool" might be replaced with a descriptor specific keypool, but I don't think that changes anything. Unless we stop using hardened derivation at the address level.
< wumpus>
jamesob: please review #11082 :)
< gribble>
https://github.com/bitcoin/bitcoin/issues/11082 | Add new bitcoin_rw.conf file that is used for settings modified by this software itself by luke-jr · Pull Request #11082 · bitcoin/bitcoin · GitHub
< gmaxwell>
in any case, changing wallets doesn't change what network you're on, so changing txfee settings doesn't really follow logically.
< sipa>
provoostenator: yes keypool will go away as well
< wumpus>
gmaxwell: agree
< luke-jr>
gmaxwell: maybe if you're worried the wallets will get linked by behaviour?
< gmaxwell>
provoostenator: keypool was configurable in the first place because it interacted with backup durability.
< jonasschnelli>
gmaxwell: I guess they want a SW wallet that derives P2SH(P2WPKH) and and their old wallet that derives P2PKH... (and not mix them)
< jamesob>
wumpus: roger that
< provoostenator>
I do like the idea of giving config / rw_config wallet specific sections.
< wumpus>
scoping settings on the wallet will be confusing for users, should imo be avoided if possible
< gmaxwell>
provoostenator: so it's not clear to me that forward caching of keys would be configurable in the future, or at least that it would be anything but an advanced thing that users usually shouldn't mess with.
< wumpus>
if anything it's very difficult to come up with a good user interface for that
< sipa>
i think all of those can either be reasonably done at the node level, or turn into descriptor-specific things in the wallet (and thus not need config)
< provoostenator>
gmaxwell: ah I see, making it "just work" could make sense
< wumpus>
every time it's the same discussion, though I haven't heard the idea of settings in the wallet seriously proposed for a while :)
< gmaxwell>
to the extent that there are wallet specific things they probably should be in the wallet so they move with the wallet. Regardless, they don't need to be part of the rwconf discussion.
< provoostenator>
Agree regarding getting rwconf out the door without adding anything to it.
< jonasschnelli>
wumpus: since its only addresstype that may be relevant and will go away over time,... I take back the argument that settings per wallet are relevant
< wumpus>
jonasschnelli: thank you
< gmaxwell>
(I'm just doubtful that there actually are more than a couple wallet specific things, addresstypes seem the most realistic of the ones mentioned here)
< provoostenator>
Indeed, that seems the most important thing to keep consistent per wallet for privacy reasons.
< jonasschnelli>
boolean things like disable-privatekeys can be handles with the 64bit wallet flags
< jonasschnelli>
*handled
< provoostenator>
One day everyone will send to bech32 addresses, and then we'll do v1 SegWit to start the problem over again :-)
< sipa>
jonasschnelli: even that we don't need post descriptors
< wumpus>
at the least, settings that are part of the wallet *should not* be part of the global options sytem
< wumpus>
otherwise it just becomes too many levels
< sipa>
you'd just not a have a descriptor with private keys if you don't want private keys
< gmaxwell>
provoostenator: no, because v1 segwit doesnt' change the address type, bech32 already supports it.
< gmaxwell>
(shocking, we already anticipated that problem... :P )
< luke-jr>
gmaxwell: but p2sh^2 could (independnetly of segwitv1)
< gmaxwell>
provoostenator: I'm sure there will be some broken sites, but it should be much less of an issue.
< jonasschnelli>
sipa: is that similar of saying "you don't need to import private keys if you don't want to have private keys"?
< jonasschnelli>
disable-private key seems to be another layer,... a footgun protection. But seems like we are going OT
< sipa>
jonasschnelli: it would turn no-private-keys into a flag for wallet creation perhaps, to determine what descriptors a new wallet is born with; but it wouldn't affect anything later
< provoostenator>
gmaxwell: but you still don't want to e.g. consistently use or not use schnorr, so you still need to maintain a preference. Even if it's all bech32.
< provoostenator>
*do want
< luke-jr>
so back to rwconf.. <.<
< sipa>
anyway, sure - we could keep it as a footgun protection, but it seems kind of pointless
< gmaxwell>
provoostenator: yes but thats just a property of which keys/descriptors are in the wallet.
< sipa>
yeah, getting offtopic, sorry
< provoostenator>
gmaxwell: good point
< wumpus>
I think it's time to move to the next topic
< provoostenator>
Descriptors are very useful.
< meshcollider>
sipa: I assume you'd still need a way to fail if you try and import private keys though
< luke-jr>
for Linux, we just support latest stable distros, so if we mirror that for Windows, we can move to Win10 :P
< sipa>
it was also one of the least popular windows releases...
< wumpus>
so I think dropping support for Vista is pretty non-controversial
< wumpus>
yes
< wumpus>
it wasn't even popular at the time
< achow101>
+1
< jonasschnelli>
ack
< meshcollider>
Yep
< sipa>
so i think there is no real to keep it
< provoostenator>
I'm a Mac fanboy, so conflict of interest :-)
< sipa>
i actually expect less opposition than the opposition we got from dropping XP support
< gmaxwell>
People are still using XP, I think less so than vista.
< luke-jr>
sipa: this may be the first time we actually break XP, note
< wumpus>
luke-jr: W7 is still used quite a lot by people unwilling to move to W10
< gmaxwell>
er more so than vista.
< gmaxwell>
the bigger issue will be W10 which is kinda spyware land windows from what I'm told.
< wumpus>
yes
< meshcollider>
Most people consider W7 as a strict improvement over vista so there should be minimal resistance :)
< sipa>
is there anything between w7 and w10?
< luke-jr>
8.1
< wumpus>
there's a windows 8 but I don't think it ever caught on much
< sipa>
ah yes
< sipa>
anyway, ack dropping vista support
< wumpus>
ok
< sipa>
based on what i read; not a windows user myself
< gmaxwell>
that PR will it make it actually not work when it does otherwise?
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #14953: test: Make g_insecure_rand_ctx thread_local (master...Mf1812-testThreadLocal) https://github.com/bitcoin/bitcoin/pull/14953
< MarcoFalke>
Should fix the thread sanitizer issue ^
< gmaxwell>
That isn't what we mean by supported on other platforms, on linux if you want to try running on really old stuff, you can try but you're own your own.
< wumpus>
yes it sets the minimum API level so that it won't start in <W7, and makes it possible to use newer APIs
< * luke-jr>
pokes MarcoFalke
< gmaxwell>
ah it gates newer APIs. Is there a reason to not announce we don't support vista, but only bump that flag when we actually go to use one of those apis?
< gmaxwell>
(just blocking software where it otherwise works doesn't really feel in the spirit of open source)
< wumpus>
you're pedaling back now?
< luke-jr>
gmaxwell: IIRC some PR wants to use Win7 API
< gmaxwell>
luke-jr: okay.
< wumpus>
FWIW qt is also going to break vista support
< luke-jr>
I did suggest making the change in *that* PR..;
< wumpus>
this is for 0.18
< wumpus>
which is still a few months away
< gmaxwell>
wumpus: no wumpus, I'm not "pedaling back" but the logic given above that we don't support older linux doesn't actually apply to actively breaking older windows.
< gmaxwell>
If it's going to be broken by other changes in any case, thats fine then.
< luke-jr>
eg, ionice_win bumps the Windows API version too, but wasn't merged yet
< wumpus>
gmaxwell: fair enough but luke-jr was talking about supporting *only* windows 10
< wumpus>
which is kind of extreme
< wumpus>
W7 as bottom should be in line with all modern software
< luke-jr>
wumpus: just mentioning it as a comparison to how we handle Linux distros ;)
< gmaxwell>
from all I've heard running bitcoin on windows 10 sounds like it's probably a bad idea in general.
< meshcollider>
From the PR, "#14881 is using inet_pton and it's only for Vista or later. So I create this PR just for that."
< gmaxwell>
wumpus: okay, concern withdrawn. I just thought it would be sad to gratitiously break it, but since we have a reason to to do, good to go.
< luke-jr>
gmaxwell: but only Windows 10 is actually supports anymore IIRC
< wumpus>
luke-jr: yes, I know it was not seriously, but would be in line with what we do for wsay, MacOSX, but mac users like upgrading a lot more than windows users
< luke-jr>
supported*
< wumpus>
luke-jr: what does microsoft still officially support? only W10? are you sure?
< * luke-jr>
double checks
< luke-jr>
Win 8.1: Mainstream support ended on January 9, 2018
< luke-jr>
apparently you can pay for support until 2023
< achow101>
windows 7 will have security updates until jan 2020
< wumpus>
great
< luke-jr>
maybe I misunderstand extended support
< luke-jr>
Win 7: Mainstream support ended on January 13, 2015. Extended support until January 14, 2020 (January 2023 for Professional and Enterprise, users will need to pay for security updates).[5][6]
< wumpus>
so w7 is still more or less relevant
< achow101>
yes
< achow101>
and i'm pretty sure a lot of people still use it too
< jamesob>
I've talked to a number of companies who're clamoring for an address index and we've got four attempts buuuut
< jamesob>
sipa has potentially disabused me of the notion that an addr index is a legitimate approach to just about anything that isn't debugging or analysis
< jonasschnelli>
jamesob: you mean unspent address index?
< jamesob>
sure, or a more general script descriptor index
< jonasschnelli>
I agree with sipa
< achow101>
jamesob: for what are the companies trying to use an address index for?
< wumpus>
an address index over the full block chain never was a good idea, one over the utxo set was considered but never made itin yet
< jamesob>
in any case, I think we should have some sort of recommendation (and ideally a maintained piece of software) to recommend to the folks who want to do addr-indexy thing
< jamesob>
s
< luke-jr>
jamesob: "don't do that" :p
< gmaxwell>
I think unspents indexes are fine, generally, but shouldn't be a priority for the project over other concerns.. if someone wants to come and do the work for them, and to do them well, great.
< jamesob>
achow101: afaict mostly watching for activity on various addresses
< provoostenator>
Always happy to test and review those address index PRs. I think that on top of the new index scheme it's not too bad to maintain.
< luke-jr>
what if we have a full TXO/script index, and only expose it in the GUI as a block explorer? (ie, no RPC to be abused)
< gmaxwell>
jamesob: our general recommendation for watching is to import them into a watching wallet.
< wumpus>
*watcing activiity* shouldn't require an index of everything over the whole block chain
< provoostenator>
Or we could come up with a plugin system for indexes, though that's also a can of worms.
< jamesob>
jonasschnelli: indeed, as well as projects like electrs
< wumpus>
wasn't there some external project addressing this?
< jonasschnelli>
I think we should not load a complete address index on the shoulders of Core
< gmaxwell>
jamesob: and I think if there are limitations on importing for watching, we'd like to improve those.
< luke-jr>
provoostenator: I thought we had that now
< sipa>
gmaxwell: yup!
< wumpus>
it's a *huge* database in any case
< jonasschnelli>
But I agree, externally makes it a bit more complex
< jamesob>
in the next few weeks I'm going to be getting in touch with companies to get a more concrete idea of the usecases, so I'll report back with what I find
< provoostenator>
luke-jr: have what? People can compile their own client of course, but that's pretty high bar and can easily lead to rebase nightmares.
< wumpus>
bitcoin core is not meant as a chain analysis platform
< luke-jr>
wumpus: I did it in ~2 GB a year or so ago, IIRC
< wumpus>
you can do forensics using your own tools
< jamesob>
wumpus: very much agree
< gmaxwell>
jonasschnelli: right the problem for externally, is that implementing blockchain consistency is non-trivial and in my expirence most people who want to build an index don't want to deal with that.
< gmaxwell>
jamesob: more information on the actual usecases would be very helpful.
< wumpus>
it's non trivial but not rocket science either
< jonasschnelli>
Sadly people expect fast BIP44 recovery (incl. history). This seems to be the most prominent real-world usecase for an address index
< wumpus>
it shouldn't be that anything non-trivial needs to end up[ in bitcoin core
< wumpus>
we're not the world of non-trivial solutions
< gmaxwell>
jonasschnelli: most (I believe most, many at least) electrum servers don't do history, so I'm not entirely clear on the including history part of your comment.
< provoostenator>
Let's also not forget to update #14053 with concept ACK / NACK so people don't waste time on trying it if it's undesirable.
< sipa>
wumpus: blockstream.info uses electrs, which seems to work very well
< gmaxwell>
wumpus: I think the process of learning enough to do it well does a pretty good job of convincing you that you don't want it.
< wumpus>
gmaxwell: heh
< provoostenator>
jonasschnelli: BIP44 recovery can be handled once we have descriptor support for importmulti and slightly more sane behavior (or a replacement for) the keypool.
< wumpus>
sipa: that's a good suggestion then!
< gmaxwell>
provoostenator: history recovery requires a scan of the chain, or a phenominally expensive index.
< jonasschnelli>
provoostenator: you can't recover the history without an address index or a complete rescan (which is not possible for pruned peers)
< provoostenator>
Ok, rescan for pruned nodes isn't there yet.
< gmaxwell>
jonasschnelli: note that an index is also not compatible with pruning. :P
< wumpus>
alrady had it bookmarked apparently :)
< jonasschnelli>
The only light here is the scantxoutset where you can recover within seconds but not the spent-history
< provoostenator>
Though you could use the neutrino filters for that and then refetch the relevant blocks.
< jonasschnelli>
gmaxwell: it could be,... if we just keep the pointers and load the data via p2p once requested
< gmaxwell>
jonasschnelli: I think we'd get a lot more bang for our buck working on making 'wallet without history before x' (pruned wallet) a first class supported thing.
< jonasschnelli>
gmaxwell: completely agree on this
< gmaxwell>
jonasschnelli: transfering 200GB of data to do the input is not really reasonable...
< gmaxwell>
(and then not saving it... this would be pretty harmful to the p2p network if people were actually patient enough to use it. :) )
< jonasschnelli>
gmaxwell: I mean there is a PR that enabled txindex with pruning... and fetches the tx over p2p once requested outside the kept block area
< jonasschnelli>
which is slow.... but for debugging proposes okay
< luke-jr>
(this makes me think again of the concept of prune-to-slow-storage where you can plug in a USB drive when/if you need old data..)
< gmaxwell>
jonasschnelli: oh, I didn't recall that PR.
< wumpus>
luke-jr: prune-to-tape *hides*
< provoostenator>
luke-jr: slow storage being the internet? :-)
< gmaxwell>
jonasschnelli: bip 157 filtering could be used similarly.
< gmaxwell>
jonasschnelli: e.g. saved the bip157 filters locally, and scan against them.
< jonasschnelli>
yes... less disk space, more time spent on filtering... but same idea
< luke-jr>
hmm
< luke-jr>
what data do they want returned from the index? maybe we don't need to retain/fetch the block
< luke-jr>
eg, if the client would be happy with a list of block numbers or txids..
< jamesob>
are any of the BIP157/158 PRs on the high prio list? if not, they should be
< gmaxwell>
wumpus: sadly, freeking tape costs about as much as HDDs today... LTO7 tapes (6TB) cost about $70.
< wumpus>
gmaxwell: ouch
< provoostenator>
So the wallet recovery flow would be: add descriptors, download BIP-157 filters (if you didn't keep them), process filters, fetch a few blocks. User can immediately use their wallet to receive and send.
< wumpus>
gmaxwell: I guess they're more reliable though
< luke-jr>
jamesob: making light wallets stronger has a negative impact on Bitcoin :/
< gmaxwell>
provoostenator: I don't really think thats a viable long term workflow either... rather if you couldn't afford the space to keep them you probably don't want the time/bandwidth to download them.
< sipa>
luke-jr: BIP158 helps our local wallet too
< provoostenator>
luke-jr: not making them possible sends most people to web wallets, not to full nodes.
< jamesob>
luke-jr: better light wallets might help adoption
< gmaxwell>
luke-jr: I'm only interested in it for the local wallets.
< luke-jr>
jamesob: better to not have that adoption..
< luke-jr>
sipa / gmaxwell: yes, that's a point
< provoostenator>
gmaxwell: recovery should be a rare thing anyway, I assume it only happens after a disaster. In which case you'd probably download the whole chain again anyway.
< sipa>
provoostenator: it should be.
< gmaxwell>
jamesob: history has shown otherwise, bip158 doesn't make lite wallets fundimentally more usable than they are now. They're still massively worse than server driven wallets like electrum or web wallets.
< gmaxwell>
provoostenator: right but thats also a reason that fetching them when you don't have them isn't that interesting, IMO.
< jamesob>
good point - but I guess if existing light wallets switched to bip157 it'd at least ease load on existing full nodes
< gmaxwell>
jamesob: what light wallets?
< provoostenator>
gmawell is that _because_ of bip158 or just because there aren't that many developers working on light (non web, non electrum) wallets? That could change over time.
< wumpus>
1 minute to go
< jamesob>
anything using bloom-filter-based SPV
< gmaxwell>
jamesob: at least connections I see there is virtually no acutal usage of p2p lite wallets anymore.
< wumpus>
please wrap up the meeting
< jamesob>
I'll report on any compelling usecases I find for addr index, but I suspect sipa et al are right that that's usually just the Wrong way
< gmaxwell>
provoostenator: P2p lite wallets that scan the chain just end up with a very poor user expirence.
< jamesob>
in the meantime I recommend we give concept ACK/NACK to outstanding PRs which are related
< gmaxwell>
and that doesn't really have anything to do with bip158 vs bip37.
< gmaxwell>
(in fact BIP158 is somewhat worse, but slightly less of a privacy disaster)
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Dec 13 20:00:37 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< provoostenator>
gmaxwell: I was thinking mostly about Lightning wallets that rely on a lightweight wallet but don't have support for on chain (other than topping up channels). Those don't need to care about unconfirmed transactions.
< wumpus>
jamesob: I think poeople have become tired of arguing, or even explicitly NACKing those things
< gmaxwell>
(it's worse because it takes a client more time to scan the chain than with BIP37, as it has to get quite a bit more data)
< gmaxwell>
wumpus: +1
< jamesob>
wumpus: makes sense. I'll come up with a reply to both PRs that tries to explain why they're not a fit for core
< wumpus>
at least I have, I mean, if you've been in this since 2011 or so, and have to keep telling people something is abad idea while it becomes ever a worse idea (due to block chain growth)
< jamesob>
and maybe we can think about adding something to the developer guidelines about indexing
< gmaxwell>
UTXO-iyx indexes I generally concept ack (at least if they're ones that aren't specific about 'addresses' but work for spk's generally), history ones, I have NAKed.
< wumpus>
gmaxwell: yep
< sipa>
provoostenator: yes, i think BIP157 may be useful for a new class of clients that may become popular
< jamesob>
gmaxwell: so a scriptpubkey index on UTXOs is worthwhile? is that a generally useful index to have for use by core?
< gmaxwell>
jamesob: we have a problem that there aren't good instructions on watchign without an index (and we likely have bugs and limitations that make it unattractive that haven't been adequately reported).. And for the analysis usecases what people want is an analysis platform, which we're not going to provide but don't really have much to recommend.
< jamesob>
gmaxwell: agree, I don't want to spend my time on the analysis usecases
< sipa>
jamesob: i dislike UTXO index equally, but it's certainly less of a burden
< gmaxwell>
jamesob: It's both useful (esp if we gain pruned wallet support-- but also scantxoutset speedup), but also it's not incompatiable with scalablity.
< jamesob>
good point
< sipa>
gmaxwell: assuming having a local UTXO set is forever
< gmaxwell>
jamesob: it's my personal belief that within 10 years most users won't ever bother fetching the far back history (or only doing so as a background test and not keeping the results).
< gmaxwell>
sipa: yes, well when otherwise becomes viable my thinking may change!
< sipa>
(which at least for the foreseeable future it will be)
< jamesob>
#utreexo :)
< gmaxwell>
jamesob: and moreover, with the exception of ultra high end commercial installs, few users would have the resources or patience to fetch 15 years of bitcoin history.
< wumpus>
doesn't need to be 'forever', it's useful *now*
< gmaxwell>
jamesob: so any of indexes on that history are is trouble for the ecosystem, as the indexes are many times less viable resource wise than the history itself.
< jamesob>
gmaxwell: so we'd expect "hobbyists" to run pruned nodes, or do you see some more sophisticated compaction of chain history?
< sipa>
jamesob: providing bulk access to sequential data is *cheap*
< sipa>
i expect nearly everyone to not bother with having the full chain
< gmaxwell>
jamesob: My expirence is that commerical parties are often less tolerant of resource usage and more willing to accept "query blockchain.info" than hobbists.
< wumpus>
UTXO db=sophisticaed compaction of chain history
< gmaxwell>
jamesob: so I think you probably have that part backwards. :)
< jamesob>
wumpus: right but you have to validate the construction of a utxo db...
< gmaxwell>
jamesob: I expect that in the future nodes will download a historical UTXO set, verifying it against an assumevalid in their code, and then sync forward from that.
< wumpus>
jamesob: oh sure if you really want to bypass validation
< sipa>
(or in the even further future, in a utreexo or magical accumulator world, not even have that UTXO set...)
< gmaxwell>
sipa: yes, but so far all those proposals are too slow to use or increase bandwidth 10x fold, which isn't a winning tradeoff.
< jamesob>
gmaxwell wumpus: okay so that leads back into the discussion of some kind of utxo set commitment
< wumpus>
jamesob: you can have my UTXO set if you want, no guarantees though :<
< jamesob>
wumpus: ha generous
< gmaxwell>
jamesob: no, it has nothing to do with a utxo set commitment in fact.
< sipa>
gmaxwell: utreexo is at 2x bandwidth or so
< sipa>
(according to simulation results i heard)
< jamesob>
gmaxwell: you're talking about "assumevalid" in the existing sense and not an assumevalid on a utxo hash, right?
< provoostenator>
What does "utreexo" stand for again?
< gmaxwell>
jamesob: a utxo has, but "utxo commitment" has generally ment putting it in blocks which is irrelevant here.
< sipa>
unfortunately it needs blocks that commit to the utxo *proofs* (not the root) for DoS resistance
< sipa>
imo
< jamesob>
gmaxwell: so what you're saying is that 15 years from now, you think that we'll have a hardcoded hash of some historical utxo snapshot (a la assumevalid) and users will start their chain validation from there?
< gmaxwell>
jamesob: doing an assumevalid merely needs a hash, which we already have... though it would be better if nodes could record their hash for each block to make review easier.
< gmaxwell>
jamesob: Yes, short of breakthroughs in ZKPs I don't think anything else will be viable.
< gmaxwell>
We needed to constrain blocksize to a fraction of the current size to avoid time to initial sync constantly growing up even assuming computers/bandwidth/disk improving by 18% or whatever year over year. Since that isn't the case the validation time will continue to grow.
< jamesob>
gmaxwell: makes sense
< gmaxwell>
And it's already large enough that it pushes many companies to prefer hosted APIs... (even if they don't mind the initial sync time, the risk of having to take it again while the service is down is pretty unattractive). Some end users are less tolerant of loading delays, some more.
< gmaxwell>
And in any case, the same security argument for assumevalid-scripts would apply: if someone can get away with making really obvious bad changes to the code you're running, all bets are off.
< gmaxwell>
Main gaps in implementation now are that it would be really good to make it easier to audit an assumevalid hash, which maybe implies adding an incrementally updatable hash.
< gmaxwell>
And then just storing and fetching snapshots for the p2p protocol.
< jamesob>
i.e. sipa's rolling MUHASH
< sipa>
i'm leaning towards the ECMH approach now, as it's easier to cache for prevalidated txn etc
< sipa>
it's more CPU, but in generally CPU time that can easily move to separate threads etc
< luke-jr>
will it upset anyone's reviewing if I rebase #11082 now?
< gribble>
https://github.com/bitcoin/bitcoin/issues/11082 | Add new bitcoin_rw.conf file that is used for settings modified by this software itself by luke-jr · Pull Request #11082 · bitcoin/bitcoin · GitHub
< gmaxwell>
I don't disagree, however for the 'sync from a snapshot and everyone can validate the hash', we could do something more boring and just compute a conventional hash of the utxo set in the background every 25000 blocks.
< gmaxwell>
There is no particular need to know the hash at every single block... sync logistics mean that it wouldn't be realistic to support syncing from an arbritary block anyways.
< gmaxwell>
The other issue with using muash/ecmh with a sync is that it doesn't really lend itself to incremental validation of fetched chunks. So it would probably need to be used in addition to another hash in any case.
< gmaxwell>
incremental validation of chunks-- I mean you're gonna download the 2gb utxo set from 8 peers... okay, great, you get the whole thing and the echm doesn't match. What now? You don't even know what peer gave you the bad data.
< luke-jr>
btw, it's not too late to reduce block sizes; if we get an organized dev-supported proposal following Lightning going production-ready, I think we can get enough support
< gmaxwell>
luke-jr: it is too late because it already takes much larger to sync the chain than many people will tolerate.
< luke-jr>
gmaxwell: we can catch up with time
< gmaxwell>
Also, the harm from the significant reduction required would be considerable.
< luke-jr>
not post-Lightning
< gmaxwell>
luke-jr: yes post lightning too, since channel creations/closers still use capacity.
< luke-jr>
less than we need
< gmaxwell>
So you probably want a tree hash so that you can have each peer prove that the chunk they're sending you is consistent with the download you're expecting.
< sipa>
luke-jr: i don't expect there to be a "lightning going production-ready"; there may or may not be a phase where it grows sufficiently mature, and there may or may not be a phase where large scale adoption happens
< sipa>
don't put all eggs in one basket
< gmaxwell>
luke-jr: I don't think it's at all obvious that the result is less than we need.
< luke-jr>
one basket is better than none
< gmaxwell>
Current blocksizes are perfectly survivable if we get out of this fetch the whole history on every install mode.
< luke-jr>
I don't think changing the security model is an appropriate avenue for that.
< gmaxwell>
I don't agree that its a meaninful change to the security model, but even if it were, it would be a good tradeoff.
< gmaxwell>
having users and businesses not willing to run nodes because bringing one up takes days of syncing is a change to the security model.
< gmaxwell>
unambigiously.
< gmaxwell>
and we're already there, and no reduction will eliminate that. (you're right that assuming (cough) continued computer speedups it would eventually get better if blocks were significantly limited today, but even that would take a long time)
< gmaxwell>
luke-jr: not to mention that the political viablity of a decrease is essentially 0, -- something you always knew, as that was also the reason that "the limit can be removed and restored if there are issues!" was a dumb idea on arrival.
< luke-jr>
it's not zero. It just requires more education, less block space need (ie, Lightning), and ideally developer support.
< gwillen>
given the massive political consequences of merely _not raising_ the size, I can hardly even imagine the idea of lowering it being taken seriously.
< luke-jr>
gwillen: the trolls are off having fun with their altcoin now, no need to appease them anymore (in fact, they will probably encourage it because they think it's a bad idea)
< gmaxwell>
The fundimental issue that many people can't wrap their heads around a limit having any purpose at all still remains. Also the issue of it being totally unclear how much is enough.
< luke-jr>
gmaxwell: that's where education comes in
< gmaxwell>
Education only goes so far.
< gwillen>
you've proposed reductions in size before, and I do not have the sense that they have ever gotten traction
< gmaxwell>
in any case, the system inherently resists changes, so the deck is stacked against you... this is a good thing usually.
< gwillen>
even among people opposed to increasing it
< luke-jr>
gwillen: not really
< gmaxwell>
luke-jr: I don't personally see a need or use to decrease it now.
< luke-jr>
gmaxwell: because you've given up, it sounds like
< gmaxwell>
Improvements to initial sync are required regardless! and I don't see them as regretable.
< luke-jr>
real improvements to initial sync only work in combination with a block size decrease
< gmaxwell>
and with them I'm not aware of an argument as to why the current size is particularly problematic, we have _radically_ improved things like block propagation.
< gmaxwell>
(for example)
< gmaxwell>
and with the work on tx relay improvements we'll be able to cut relay overheads by maybe 40x.
< luke-jr>
that's unrelated to initial sync..
< gmaxwell>
exactly, I am pointing out that the current size is okay except for initial sync. And initial sync needs to be safely shortcutted regardless of the ongoing future size. And once it is, initial sync would no longer be a huge issue.
< luke-jr>
I don't agree. That's what I mean by just giving up.
< gmaxwell>
unless you think that downloading bitcoin node software written by other people and not doing the work of writing it yourself is giving up. :P
< luke-jr>
code can be verified
< gmaxwell>
with an extreme amount of work, and an AV can be verified, with considerably less cost/work than code.
< luke-jr>
not if we give up on a full sync
< gmaxwell>
I'm confused by that view.
< gmaxwell>
set -assumevalid=0 and you'll not use it, you'll take a lot longer to sync. It's still radically cheaper than doing pratically any code audit. (especially since in the common case of someone bringing up a node when they already had one running, their review need only be to compare between their nodes)
< luke-jr>
gmaxwell: but we're talking about leaving in place an infinitely growing initial sync time
< luke-jr>
right _now_ it's easier, but it won't be for long
< gmaxwell>
The cost of an outsider finding a subtly introduced 'bug' in the code is phenomial, maybe the cost of a review that could reliably catch something like that is on the order of $100,000 ... and that cost goes up over time as wages for experts seem to go up and the codebase increases in size.
< gmaxwell>
It would take a LONG time of growth before the cost of a non-AV sync matches the cost of review that would reliably find a subtle bugdoor.
< luke-jr>
therefore everyone should just trust developers /s
< gmaxwell>
I don't see how that follows.
< gmaxwell>
Verification is costly, there isn't any way around that.
< luke-jr>
it's something people can still do today, for the blockchain
< gmaxwell>
luke-jr: and they will still be able to do it in 10 years at current growth rates. it will just continue to be burdensome enough that most people will choose not to do so, exactly as is the case for the rest of the code today.
< gmaxwell>
Worrying that it'll take 40 hours of computer time instead of 8 seems penny-wise-and-pound foolish, when it'll take 40 hours of _human_ time to do even a light review of the code.
< gmaxwell>
in terms of aggregate validation, political efforts would probably be better spent making implementations more verifyable.
< treyzania>
inb4 bitcoin node in coq
< aj>
fwiw, with the "do nothing" approach, assuming IBD speeds increase by 18% per year, and chain length grows by 200GB/year, IBD maxes out at around 2.6x current length in 5 years, and finally gets back to current time after ~18 years
< gmaxwell>
aj: thanks.
< aj>
oh, except 100GB/year is more reasonable? 2MB * 1000 blks/week * 52 weeks/year?
< aj>
that's max of 1.5x in 4 years, back to same time as now in 12 years
< phantomcircuit>
gmaxwell, the only reason i can think you'd want an addrindex is if you have a pile of private keys and cant remember any metadata about them
< gmaxwell>
phantomcircuit: even there, do you really want an addrindex or a utxo spk index? Also if it's actually a pile, scantxout set will be faster.
< phantomcircuit>
gmaxwell, for whatever accounting reason you need the history not just the current utxo
< aj>
decreasing the weight limit to 2Mweight under those assumptions (so 50GB/year) keeps IBD time increase under 10% from now
< gmaxwell>
phantomcircuit: indeed. But for that case, you'd still probably be just as well off with bip157 filters on disk, and sequential scanning those.
< gmaxwell>
well it's not an option yet. yea, without more inforation on what people are asking for I'm not sure if people would be satisified by disk-filter powered rescan.
< phantomcircuit>
gmaxwell, im 99% certain what people want is an easy way to import historic payments to an address for accounting reasons
< phantomcircuit>
and have that work with whatever custom wallet stuff they've written
< phantomcircuit>
(ie no creation timestamp)
< gmaxwell>
that absoluely is a reason, I doubt its the only one.
< phantomcircuit>
im sure the other 1% is blockexplorers calling rpc directly
< gmaxwell>
e.g. like I've talked to an exchange who was asking for an index because they thought it would make the daemon into a block explorer and they could use it to answer 'sent from' for blacklisting purposes.
< gmaxwell>
But none of the addrindex proposals would have made that possible, regardless...
< gmaxwell>
because they index outputs, not inputs.
< adiabat>
sorry just got to this but a couple maybe relevant points:
< kanzure>
late hi
< adiabat>
I'm hoping the utreexo thing will help clients sync faster, even without any checkpoints. If you're running on an HDD, disk I/O is the main delay for IBD
< adiabat>
and with utreexo / other utxo accumulator, you can avoid touching the disk and stay in ram, so you are just cpu limited by signatures
< adiabat>
(which fast computers with SSDs are now anyway, but lots of people run nodes with spinning drives)
< adiabat>
also separate point is that the need for a bridge node; these nodes can attach proofs to inputs for the nodes which hage pruned the utxos set away and only hold the accumulator
< treyzania>
I wonder how long it would take to sync off of a bunch of RAIDed floppies
< adiabat>
the bridge node needs a utxo index; not of addresses though. But I'm not sure if it makes sense to build that into core or keep it as a separate server / node / software
< adiabat>
so that might be a place to put address index software if people don't want it in bitcoind
< luke-jr>
oh wow, I totally forgot about adiabat's thing after Tokyo >_<
< adiabat>
yeah I need to put stuff up publicly. I've been working on it but it takes longer than I would expect, like evertyhing else :)
< instagibbs>
(btw he quoted 15% overhead during IBD in other channel)
< adiabat>
instagibbs: yeah, overhead depends on how much ram you dedicate to caching the proofs
< adiabat>
with no caching I have 250GB of proofs, so not great
< adiabat>
but it goes down quickly with increased caching. Also I have a maybe controversial way to use shorter hashes.
< luke-jr>
did kanzure transcribe the discussions in Tokyo? I feel like I should review them now :/
< adiabat>
in that you can argue that you're only needing 2nd preimage resistance, not collision resistance. But I'm not 100% set on that, even though I'm pretty convinced
< gmaxwell>
adiabat: you can sync entirely in ram today, it just takes 8GB of cache... so you can measure the performance impact of that vs defaults.
< gmaxwell>
adiabat: it's non-trivial, but I am not expecting that it would be better for sync time to double bandwidth usage but avoid the in memory state.
< sipa>
gmaxwell: with a couple hundred MB of cache it seems the overhead is just 15%
< sipa>
if that's true, it starts to sound appealing for far more deployments
< gmaxwell>
well currently.
< gmaxwell>
I'm confused about that, it seems improbable to me. 15% would mean only on the order of _one_ extra hash per input.
< * sipa>
looks at adiabat
< gmaxwell>
2x I believed. :)
< treyzania>
it works *really* great for embedded devices which might have a few hundred MiB memory and flash storage
< treyzania>
or like rpis
< adiabat>
gmaxwell: it's because so many utxos are short-lived, and this is with relying on "hints" from the archive node you IBD from
< adiabat>
the archive node knows what's happening next and can give TTL values for all the utxos created in a block
< gmaxwell>
Oh I see the few hundred mb cache is basically a cache of recent outputs, plus the top of a tree for non-recent outputs (or similar)?
< adiabat>
better than recent -- upcoming
< sipa>
adiabat: but that doesn't apply for chain tip updates, only historical sync
< adiabat>
that part is trusted, so they could lie and say long-lived utxos will be consumed soon
< adiabat>
right, this is only for IBD
< sipa>
i think the affect on bandwidth to keep in sync is more important
< adiabat>
these optimizations don't help once you're synced up; I've mostly been focusing on IBD so far
< adiabat>
there are other ideas for in-sync, but I haven't worked on that much; with IBD I have a nice data set to work off of to get performance numbers
< adiabat>
archive nodes indicating which utxos will be consumed soon could potentially be useful for IBD with the current levelDB as well I guess
< adiabat>
worst case if the node you're downloading from lies to you, your cache is full of the wrong things and you have to go to disk more and hang up on the lying node
< gmaxwell>
I don't think this is currently an interesting route of optimizations, from an engineering perspective. Right now the existing software doesn't implement a simple fifo for management of the cache, in part due to the complexity of doing so. Fifoing inputs would probably have 98% of the locality gain (a question I guess your research could answer), but still less complexity than cache
< gmaxwell>
management hints.
< adiabat>
yeah, fair, it makes more sense with the accumulator proofs in that you prevent having to download a lot of data
< gmaxwell>
Right sounds more useful there.
< gmaxwell>
But even there in IBD you're essentially trading off (say) 200gb * .15 = 30GB more data transfered, in order to avoid having to store 2GB durign sync. I think under most relaistic models of the relative costs of storage and bandwidth, this tradeoff isn't that amazing. Having the option of making it would be nice, but I would expect relatively few systems to actually be better off with that
< gmaxwell>
choice.
< adiabat>
for powerful computers it doesn't really help, but for computers with HDDs / space constraints it can help
< sipa>
or with I/O-starved VPSes
< sipa>
and perhaps it's also just faster
< adiabat>
but yeah if you have a fast computer and slow internet, it makes things worse
< sipa>
(to be seen)
< sipa>
and it also makes assumevalid-utxo models trivial
< sipa>
(no snapshot to transfer...)
< adiabat>
well a bech32-encodable snapshot
< adiabat>
hm actually no, it's a few hundred bytes so not quite... :)
< gmaxwell>
if you're assuming that the node knows the top of the tree, in order to avoid an absurd bandwidth blowup, then you'll have to transfer that.
< sipa>
that just gets relayed along with the first transactions that need it
< adiabat>
you could tranfer the top along with the root, but as soon as you see a few proofs in the mempool, you'll get the top
< adiabat>
right, what sipa said
< sipa>
it's of course still bandwidth; just saying it doesn't need a special store-and-transfer-a-snapshot protocol
< gmaxwell>
Okay, but thats just hiding the cost.
< sipa>
by trivial i meant complexity, not bandwidth - but even there you're probably right that it's complexity we're paying anyway in a more complex block/tx sync protocl
< gmaxwell>
sipa: it does need a snapshot, because you cannot run from the tip without wreaking security, you'll need the state as of the point you know is valid.
< adiabat>
yeah, network bandwidth is the big cost to this, CPU cost is low
< gmaxwell>
(I almost agreed with you for a moment there though! :P )
< sipa>
gmaxwell: ok, sure - but it's a few hundred bytes
< gmaxwell>
really? there are 68m utxo or so, 1000 bytes would allow for 32 hashes. for 68m hashes we need a tree of depth 29. or 24 if 5 levels are saved from the top, so that results in 768 bytes per input of membership proof.
< gmaxwell>
so I think not a few hundreds bytes... certantly a lot smaller than the whole utxo set though... assuming you don't want to be able to offer proofs yourself.
< sipa>
what did i miss?
< luke-jr>
pizza
< gmaxwell>
dunno if you missed anything.
< sipa>
that's okay
< gmaxwell>
did you see
< gmaxwell>
14:38:47 < gmaxwell> really? there are 68m utxo or so, 1000 bytes would allow for 32 hashes. for 68m hashes we need a tree of depth 29. or 24 if 5
< gmaxwell>
levels are saved from the top, so that results in 768 bytes per input of membership proof.
< gmaxwell>
14:39:16 < gmaxwell> so I think not a few hundreds bytes... certantly a lot smaller than the whole utxo set though... assuming you don't want to be
< gmaxwell>
able to offer proofs yourself.
< gmaxwell>
?
< adiabat>
I'll clean up the code a bit, but the code I have is not a single tree; it's a forest of perfect binary trees
< sipa>
gmaxwell: the "state" of the utxo set is just the roots of each of the trees in the forest
< adiabat>
so the worst case proof yeah is like 28 high
< sipa>
it's distinct from the cache of elements in the trees (which just gets transferred along with the blocks)
< adiabat>
but you can try to put most of the old utxos on the left in the big tree, and most proofs that you need to transfer are in the smaller trees with high turnover
< gmaxwell>
sipa ... adding 28 hashes per input is a non-starter for general usage. There are some applications, vps in a datacenter where internal bandwidth is free, where it might be attractive. but for most systems _bandwidth_ is the most limited resource.
< sipa>
gmaxwell: of course
< adiabat>
gmaxwell: 28 hashes per input is too much, agreed. There's ways to bring that number down significantly
< sipa>
gmaxwell: i think you're missing some of the ideas
< adiabat>
(which I haven't written up, so not you're fault :)
< gmaxwell>
adiabat: it can be brought down by caching more of the top...
< sipa>
but in general i think it's a semantics discussion about what you call state and what you call cache
< adiabat>
but a big part is exploiting the power-law dureation of utxos
< sipa>
gmaxwell: adiabat is well aware
< gmaxwell>
adiabat: That doesn't come about as a result of any physical law or even economic incentive in the current system. :(
< adiabat>
true, a lot of is is a heuristic, so who knows. if you have uniform random access across the utxo set, it gets worse.
< adiabat>
fitting it to the current 200GB data set we have though, it's quite consistent that many utxos persist for only a few blocks
< gmaxwell>
sipa: please we cannot continue to fail to do something about sync times in favor of conjecture about future ideas with uncertian performance. Like here "It's 2x" but only under assumptions about spending patterns, that is _really_ misleading and I think it undermines productive discussion.
< adiabat>
(many get created & destroyed in the same block which means they never even touch the accumulator)
< sipa>
gmaxwell: for the sake of discussion, please just treat the caching we're talking about as what you're calling keeping the top levels of the tree as state
< adiabat>
gmaxwell: I certainly don't mean to over-sell this; it's not a silver bullet, but I think it can help in some cases
< gmaxwell>
adiabat: Sure.
< sipa>
gmaxwell: fwiw, our current performance for sync with dbcache less than the entire UTXO set is also dependent on spending patterns
< gmaxwell>
sipa: sure. (though less than it would be if the dbcache's managment weren't kinda stupid wrt locality)
< gmaxwell>
But "oh look spending patterns changed and with no impact in fees, bandwidth needed by bitcoin nodes just went up 5x" isn't the same as saying dbcache works better with more locality.
< sipa>
yeah, agree
< sipa>
we need a separate average case and worst case analysis w.r.t. bandwidth impact
< gmaxwell>
I am frustrated because these kind of far off researchy things seem to keep getting pulled out as a reason to not do something more near term about the initial synctime problem.
< sipa>
i don't think so
< gmaxwell>
Well, that is how I saw the discussion today.
< sipa>
as far as i'm concerned, the potential prospect of utreexo or similar techniques are only a reason to not support a utxo-address index for me
< gmaxwell>
That makes sense. I wasn't connecting it to that discussion.
< gmaxwell>
We can debate on the necessity of users getting historical data except for research purposes, no such debate exists for utxo though... so some scanning mechenism must be provided even in a utxotree world. ... Ideally one that preserves user privacy. But there can be ways of doing that.
< adiabat>
fwiw I think it's towards the practical / implementable side of research-land. I certainly don't want to say "this solves everything, don't worry about sync times, I got this" though.
< gmaxwell>
adiabat: Towards perhaps, but I think you overestimate that. The gap between what we can imagine and build for testing and what we can safely put into production is pretty big. Right now Bitcoind's dbcache grows up to some limit then is flushed completely, rather than acting as a fifo queue that exploits locality. "Make it fifo" sounds trivial, but actually getting it done, and Right, and
< gmaxwell>
being confident that its right is not.
< sipa>
adiabat: well, we'll see - i think you may be underestimating the amount of work needed to deploy archive nodes, and the practically implementing a dos-resistant stateful sync protocol for parts of the utxo tree
< sipa>
adiabat: even after you've characterized all the memory and cpu requirements of a close to fully written up spec
< adiabat>
agreed, it's always harder than it seems. Also, I don't mean to put it into bitcoind, I'm going to start with more of a server / client model
< adiabat>
I guess in comparison to the class group accumulator it's more implementable though
< sipa>
wahaha
< sipa>
yes
< gmaxwell>
sipa: in any case, I was connecting this discussion back more to the assume valid discusion instead of the indexing discussion. So to me it sounded like an argument to do nothing because utxotree will solve everything... and I'm dubious that the bandwidth overheads will make it a _general_ solution (obviously a big win in some cases), and I'm dubious as to how fast it can be made available.
< gmaxwell>
adiabat: yea, funny I dunno about that! so like a class group accumulator is an isolated number theory programming project.
< gmaxwell>
so just from a software eng I think CGA might be more implementable, but it's just too slow to make useful regardless.
< sipa>
gmaxwell: my current estimate for updating an archival node with a new block is about a year of CPU time
< sipa>
gmaxwell: with class group based accumulators
< gmaxwell>
sipa: lol yes, thats my 'too slow to make useful'
< sipa>
oh of course
< gmaxwell>
but like, _ignoring that_ (lol), in terms of actually switching bitcoin to use it, I expect it would be easier than utxotree.
< sipa>
also ignoring the novel cryptographic assumption? :)
< adiabat>
huh, yeah the proofs are tiny, everything's constant sized, etc. The utreexo accumulator stuff does have a whole bunch of weird logic.
< gmaxwell>
Even including the cost of having to write a "fast" classgroup acc library (assuming fast doesn't mean anything more than well optimized)
< sipa>
but sure, everything constant size makes a lot of logistical problems go away
< gmaxwell>
including need for protocols that make roundtrips... the bane of everyone's existance.
< gmaxwell>
sipa: in any case, going back to the indexing question... for history indexing part of my concern there is that for some use cases there is a "just don't do that" or "look in the utxo instead" option. Like "We need a history index to look up from addresses!" :P for the utxo set that is much less the case.
< gmaxwell>
so if you imagine that in the future everything is using utxotree, we'll still need to provide some way to look up scriptpubkeys... maybe it's PIR queries to nodes what have an N of M shared copy of the whole database or whatever... but something will have to be provided.
< gmaxwell>
as rescan is already pretty unrealistic and contantly getting worse.
< sipa>
gmaxwell: i'm not sure what use cases really need access to the full UTXO set
< sipa>
of course there is a convenience in having it
< sipa>
but there is convenience in having an address index for the whole chain too
< gmaxwell>
sipa: I need to take my wallet offline for a long time or recover a backup and want to spend my funds.
< gmaxwell>
To actually _use_ bitcoin I need that data... vs history is needed for analysis/auditing/etc purposes.
< gmaxwell>
(and for those analysis you probably don't just want an address index in history you want all sorts of other indexes)
< sipa>
gmaxwell: assuming the UTXO set was equally expensive to look things up as in the blockchain (it's obviously not, but just as a hypothetical), would you still say that?
< gmaxwell>
yes, because you can't spend your bitcoins without the utxo data!
< sipa>
sure, but we need access to the chain anyway in recovery scenarios
< gmaxwell>
(I mean if they were equal, I suppose you could just use the blockchain instead. but the utxo part is special because it's actually needed to make any use of all of your coins)
< gmaxwell>
sipa: we don't, the fact that we recover tx history is an artifact of how the software was implemented.
< gmaxwell>
Considering we can't recover tx medatadata, its not like we're actually recovering the real history.
< gmaxwell>
The history can well be useful, for sure, but it is not needed to make use of your bitcoins.
< gmaxwell>
(and other wallets, e.g. electrum, can run without history)
< sipa>
gmaxwell: for a business knowing history may be far more important than the ability to spend coins
< sipa>
my point is that what you're talking about is pretty much a recovery scenario that should be rare; and the only reason you consider is more reasonable than seeing whole history is because it's currently (and presumably later) much cheaper
< gmaxwell>
sipa: I disagree because the thing we can recover from the blockchain is not history, it's partial history-- it lacks metadata which is actually the important information.
< gmaxwell>
moreover, that data is there-- go use a block explorer, there is no need a bitcoin wallet or node need to provide it.
< sipa>
also true; generally you also need information you'd have backed up separately
< * gmaxwell>
bbl
< lopp>
question regarding maintaining code integrity during the development process: is there any way that an adversarial maintainer could merge an altered PR commit? just trying to fully visualize the "chain of code custody" between the steps of peer review and actually being merged