< wumpus>
as it is impossible to remove approvals, random-approval-spamming trolls wouldn't be unthinkable. Note that for normal posts, the user gets changed to 'ghost' if they disappear, so the link doesn't break.
< paveljanik>
I think the approval stuff is half-baked...
< paveljanik>
So maybe someone is trying DoS on us...
< wumpus>
it's only on one issue AFAIK, so I doubt it'd be targetted on us :)
< paveljanik>
./DoS ... and then while :; do ./DoS; done ;-)
< wumpus>
or they got caught *really* early; more likely such a troll/bot would just randomly approve pulls all over the site
< paveljanik>
wumpus, search for the username on the github...
< paveljanik>
complete github...
< wumpus>
luckily they didn't add a message, the message added with review approval is also un-removable/un-editable IIRC
< GitHub131>
bitcoin/master fa05cfd MarcoFalke: [rpc] throw JSONRPCError when utxo set can not be read
< GitHub131>
bitcoin/master 7d563cc Wladimir J. van der Laan: Merge #8832: [rpc] throw JSONRPCError when utxo set can not be read...
< GitHub93>
[bitcoin] laanwj closed pull request #8832: [rpc] throw JSONRPCError when utxo set can not be read (master...Mf1610-rpcUtxoFail) https://github.com/bitcoin/bitcoin/pull/8832
< GitHub3>
bitcoin/master 64047f8 Wladimir J. van der Laan: depends: Add libevent compatibility patch for windows...
< GitHub3>
bitcoin/master 489a6ab Wladimir J. van der Laan: Merge #8730: depends: Add libevent compatibility patch for windows...
< GitHub165>
[bitcoin] laanwj closed pull request #8730: depends: Add libevent compatibility patch for windows (master...2016_09_libevent_windows_gcc_531) https://github.com/bitcoin/bitcoin/pull/8730
< GitHub162>
[bitcoin] laanwj closed pull request #7522: Bugfix: Only use git for build info if the repository is actually the right one (master...bugfix_gitdir) https://github.com/bitcoin/bitcoin/pull/7522
< wumpus>
cfields_: I've assigned some build-system issues to you, hope you don't mind
< wumpus>
weird, I'm on testnet and trying to rebroadcast a transaction using "sendrawtransaction" but it's not working, I don't see anything appear in my wireshark session
< wumpus>
maybe all the current nodes already know of it, otoh the "broadcast through 1 node(s)" in status doesn't increase either
< sipa>
could it be that your peers knew about it, but evicted it?
< wumpus>
but for they they'd have to request it first
< MarcoFalke>
What about resendwallettransactions
< sipa>
not necessarily from you
< wumpus>
MarcoFalke: this should work with sendrawtransaction
< wumpus>
could be that the net refactoring work changed the assumptions, will add some debugging...
< sipa>
i don't think so
< MarcoFalke>
its a noop when fHaveMempool?
< sipa>
no
< wumpus>
I don't hope so
< MarcoFalke>
sendrawtransaction will only put in in your mempool
< sipa>
and then loop over all peers
< sipa>
and call PushInventory
< MarcoFalke>
oh
< sipa>
but PushInventory is a noop if filterInventoryKnown for that peer already contains the tx
< wumpus>
but it can't be there unless the peer requested it from *us*right?
< wumpus>
hm or if they inved it for relay to us, I guess
< wumpus>
I don't have a capture of the whole session so we'll never know for sure
< sipa>
well you can enable -debug=net and see if any messages go out in response to sendrawtransaction
< sipa>
oh
< sipa>
you're already doing that
< wumpus>
no, no messages went out
< wumpus>
and according to the tx metadata apparently one peer ever requested the transaction
< wumpus>
but that was before I started logging
< wumpus>
it could also be that that counter is broken, of course, I don't think that functionality is tested anywhere it's UI only
< wumpus>
ok, seems the behaviour was correct. After restart the transaction is no longer in the filter, so I do sendrawtransaction again, it sends out an inv to every node.
< sipa>
i wonder if we should add some random memory/cpu intensive task occasionally during block validation (so it doesn't consume more than 1% cpu rate or so)
< sipa>
and if that fails, tell the user their hardware is too unreliable
< wumpus>
yes, some games have that, it's not a bad idea
< wumpus>
it allows distinguishing bugs from hw failures
< wumpus>
we have the same problem, 'support' is overflowed with issues that are probably hw failures but we can't be sure so it wastes a lot of time
< paveljanik>
We could ask users for the result of bitcoind -sanitychecks or something if we suspect HW issue...
< sipa>
paveljanik: the problem is that such a sanitycheck would need to run for several hours or so to be reliable
< paveljanik>
sure, but 90% rule...
< sipa>
if a failure is detectable within minutes, it also means their block validation like fails within minutes
< sipa>
and many other things
< sipa>
most random failure i hear about happen after several hours of sync
< wumpus>
the sanity check definitely needs to run automatically and periodically for it to be useful
< wumpus>
because otherwise it won't run in the right conditinos
< wumpus>
we should start selling a hardware quality certificates: has synced the bitcoin block chain succesfully
< wumpus>
if it can do that, it won't crash on games either. Well. Maybe after GPU secp256k1 verification is implemented ;)
< sipa>
maybe we should instead just market secp256k1 asics
< wumpus>
would be a very interesting project if there's a market for that
< wumpus>
hm, scrap that. THe market for that is key-cracking :(
< wumpus>
searching around a bit, apparently, many people *started* on a verilog/vhdl FPGA implementation of secp256k1, but there's no code to be found anywhere
< wumpus>
I'm pleasantly surprised how well the wireshark dissector for the bitcoin protocol works, can just do "tcp port 18333" then use a display filter of "bitcoin" and gets a running list of bitcoin packets easy to inspect using the tree structure
< wumpus>
compared to trying to mentally parse debug.log output this is a breeze
< waxwing>
wumpus: they've had that dissector for years, i remember being pleasantly surprised in like 2013/14
< waxwing>
not that i need it or anything, but it's cool :)
< wumpus>
yeah yeah it's probably not new
< wumpus>
but just in case someone didn't discover it 20 years ago yet, there you go...
< waxwing>
sorry wasn't trying to be a hipster :)
< wumpus>
:-)
< sipa>
damn. i believe this is a blocker for bip151.
< wumpus>
agree sipa
< wumpus>
though it is possible to have dissectors with parameters, such as a key, but it's so much less user friendly!
< wumpus>
a more practical way to go at this, post-bip151, would be add functionality to dump packets to disk after decryption on receive and before encryption on send, it's for debugging anyhow not snooping
< waxwing>
iirc there are per-dissector settings, for ssl you can enter the session key there etc.
< wumpus>
yes
< waxwing>
there's an env variable you can set when running firefox that dumps the keys for you, and you can import it in
< sipa>
implementing bip151 in wireshark will be fu
< waxwing>
and chrome i think
< wumpus>
waxwing: but if you deal with lots of different sessions, or sessions that are created on the fly
< waxwing>
right, which i guess might be of particular import in your case (/me hasn't read bip151 tho)
< wumpus>
it's certainly possible, the only remark I was making was in regard to what was the most practical way :)
< wumpus>
one advantage of doing the decryption in the dissector would be that it could detect crypto problems
< sipa>
i don't know what language wireshark uses... but reimplementing sha256, chacha20 and poly1305 doesn't sound trivial
< sipa>
(unless those are already available as primitives)
< wumpus>
C
< sipa>
oh, the filters too?
< wumpus>
it's also possible to write dissectors in lua, but that's only recommended for one-off projects, as performance is abysmal
< sipa>
i thought those would be in a plugin language like lua or so
< wumpus>
yes, most of them
< sipa>
ah.
< sipa>
the bitcoin dissector is in c currently?
< wumpus>
let me see
< sipa>
i vaguely remember seeing the code for it, years ago
< GitHub156>
[bitcoin] MarcoFalke opened pull request #8834: [qa] blockstore: Switch to dumb dbm (master...Mf1610-qaBlockstoreDumb) https://github.com/bitcoin/bitcoin/pull/8834
< wumpus>
apparently it's not yet updated for 0.12.0+, no handler for sendheaders, feefilter etc
< mmeijeri>
The same environment variable (SSLKEYLOGFILE) also works for Chrome.
< wumpus>
oh it's a dynamic file? that's a good idea
< waxwing>
mmeijeri: it's coming back to me, i have a feeling that it handles multiple sessions transparently, by just checking which key material works for which session. not that this matters, sorry for OT :)
< waxwing>
ah yes that was it, i think it stores the premaster secrets for all sessions, then in each handshake it tries them out until it finds what works. something like that.
< GitHub180>
[bitcoin] MarcoFalke opened pull request #8835: [qa] nulldummy.py: Don't run unused code (master...Mf1610-qaNulldummyUnused) https://github.com/bitcoin/bitcoin/pull/8835
< GitHub8>
[bitcoin] jnewbery opened pull request #8836: bitcoin-util-test.py should fail if the output file is empty (master...bitcoin-tx-no-empty-outputs) https://github.com/bitcoin/bitcoin/pull/8836
< GitHub100>
[bitcoin] jnewbery opened pull request #8837: allow bitcoin-tx to parse partial transactions (master...bitcoin-tx-partial-transactions) https://github.com/bitcoin/bitcoin/pull/8837
< GitHub16>
[bitcoin] jnewbery opened pull request #8838: Only log block size if block size is being accounted (master...dont_log_size) https://github.com/bitcoin/bitcoin/pull/8838
< GitHub17>
bitcoin/master 7e5fd71 Pavel Janík: Do not include env_win.cc on non-Windows systems
< GitHub17>
bitcoin/master f560d95 Wladimir J. van der Laan: Merge #8826: Do not include env_win.cc on non-Windows systems...
< GitHub75>
[bitcoin] laanwj closed pull request #8826: Do not include env_win.cc on non-Windows systems (master...20160928_leveldb_no_win) https://github.com/bitcoin/bitcoin/pull/8826
< GitHub51>
[bitcoin] laanwj opened pull request #8840: test: Explicitly set encoding to utf8 when opening text files (master...2016_09_textfiles_locale) https://github.com/bitcoin/bitcoin/pull/8840
< cfields_>
jonasschnelli: i had a look at your circular dep issue. No luck here. The (hack) solution is to use grouping libs, but I'd really rather not go down that road. I've started untangling dependencies instead.
< wumpus>
which dependencies are the problem there? the circular dependency between libbitcoin_server and libbitcoin_wallet?
< jonasschnelli>
topic proposal: pruning and blockrelay
< petertodd>
hi
< sipa>
policy against uncompressed keys or not
< wumpus>
#topic pruning and blockrelay
< jonasschnelli>
I think we should add a service flag for block relay with a min-height
< jonasschnelli>
NODE_PRUNENETWORK or something
< sipa>
there have been multiple ideas around that
< petertodd>
IMO whatever we do, we should recognise that w/ segwit's larger blocks we can expect a lot of full nodes to run out of disk space quite soon
< sipa>
the easiest is just to add a flag that says you relay valid blocks and transactions, but not historical blocks more than a few deep
< jonasschnelli>
I guess people slowly start to prune the blockchain to a max of 80GB or similar... but I guess not everyone is aware of the fact that you don't relay then
< wumpus>
it would be nice to support more than one range, e.g. also archive nodes that host part of the old blocks
< sipa>
it becomes harder when you want multiple ranger
< petertodd>
do we have a reason for more than one range?
< jonasschnelli>
We could introduce another message type... blockrange or so
< sipa>
it becomes even harder when you want to support sharding in an efficient way
< wumpus>
I'm not sure why itb ecomes hard, just add a query message that returns what ranges are supported
< petertodd>
sipa: what do you mean by sharding exactly?
< sipa>
petertodd: you'd configure your node to maintain a certain % of blocks
< jonasschnelli>
wumpus: query, yes, why not, or just inform like we do with sendheaders
< petertodd>
sipa: see, given that the bitcoin protocol can't be safely sharded right now, I think we can safely say that we don't need to support sharding in block relay yet
< petertodd>
sipa: doing so might even be dangerous if people start using it
< sipa>
petertodd: not in block relay
< sipa>
petertodd: for block archival
< petertodd>
sipa: but why shard vs. keep ranges?
< petertodd>
sipa: (ranges of full blocks)
< luke-jr>
BitTorrent already does this. Surely we can learn from that?
< petertodd>
luke-jr: I don't think so - bittorrent is a very different problem than bitcoin
< wumpus>
this is for letting other peers know what ranges of blocks are hosted, I don't think this should affect releay
< sipa>
so, i've been running statistics on what block depths are being requested from nodes
< luke-jr>
petertodd: learn from it, not use it directly
< luke-jr>
petertodd: BitTorrent's problem isn't very different from IBD
< jonasschnelli>
sipa: interesting.. do you have the stats public available somewhere
< jonasschnelli>
I wanted to do this a long time
< petertodd>
luke-jr: so, the thing is bittorrent has the problem of a diverse set of files, we just don't have that problem and can optimise differently because everyone needs access t othe same set of data
< sipa>
there are something like 4 meaningful 'ranges' 1) the top 2 blocks (just relay) 2) up to ~2500 blocks deep... requested very often 3) up to ~10000 deep... requested a few times more than the next range 4) the rest
< wumpus>
otoh bittorrent has a fixed block size :)
< sipa>
wumpus: so do we *ducks*
< petertodd>
sipa: that probably corresponds to how long people leave their nodes offline :)
< btcdrak>
inb4 Bittorrent XT
< sipa>
jonasschnelli: they're not available, and the ranges i gave above are just from me quickly glancing over the result
< petertodd>
btcdrak: I use Bittorrent Unlimited myself
< jonasschnelli>
What about fingerprinting issued in conjunction with available ranges?
< jonasschnelli>
*issues
< petertodd>
jonasschnelli: make them powers of two?
< sipa>
well 4 ranges can be done with 2 service bit flags
< sipa>
gmaxwell: you've worked on these ideas before, comments?
< jonasschnelli>
But would that work with the flexible pruning option based on MB?
< petertodd>
jonasschnelli: sure, just find the biggest range less than the pruning amount
< sipa>
jonasschnelli: you'd change your service bits on the fly
< wumpus>
why would the ranges need to be in the flags?
< gmaxwell>
sorry, I missed that the meeting started.
< jonasschnelli>
Yes. Why? Better add an explicit message for the range
< sipa>
how would you otherwise discover what nodes to connect to?
< sipa>
just randomly try?
< petertodd>
jonasschnelli: oh right, you mean if MB != blocks... sorry.
< wumpus>
I think you'll need a service flag to show support for the protocol, but not what ranges you have
< jonasschnelli>
query or inform the other node if proto-ver > NODE_PRUNENETWORK
< wumpus>
well that can be negotiated later, like bittorrent does I guess
< sipa>
wumpus: well you do want addr messages to contain this information
< wumpus>
I doubt bitcoin has 'service flags' in its tracker what blocks nodes have
< petertodd>
sipa: so the nice thing about bitcoin, is just randomly try will probably work fairly often due to the low number of ranges out there
< wumpus>
as that changes all the time anyhow
< gmaxwell>
I was strongly of the view that we needed to signal at least two ranges. Sipa's latest measurements make me think at least three are needed.
< wumpus>
s/bitcoin/bittorrent/
< jonasschnelli>
I think informing other nodes ranges over addr is another thing...
< jonasschnelli>
A first step would be a information after connect
< wumpus>
yes, addr is another thing
< gmaxwell>
I think ranges in service bits are no big deal, the harder question is what to do about the history. having nodes with 150GB of history in order to serve the last range is not very viable.
< wumpus>
could be done later if an efficient way is needed to *locate* peers with certain ranges
< wumpus>
but that seems premature optimization
< gmaxwell>
We will need to redo addr sometime relatively soon in any case, as our messages are not compatible with HS-NG.
< petertodd>
gmaxwell: oh, you mean Tor's new hidden services standard right?
< gmaxwell>
petertodd: yes.
< gmaxwell>
(also I2P though thats not new)
< wumpus>
I think the number of ranges should be variable
< wumpus>
redesigning addr is a different topic
< wumpus>
also necessary, but again, doesn't need to be on one heap
< gmaxwell>
wumpus: when I'm saying ranges I am specifically referring to the top-N zomes.
< petertodd>
well, so if we add service bits for recent history ranges, that should be possible to implement as a separate feature to archival history ranges, and it'd be a big first step
< wumpus>
I think it should be possible to, say, only host the first 20GB of blocks
< jonasschnelli>
historic only nodes
< wumpus>
I don't see why it should be restricted to only recent history
< petertodd>
I don't think it's likely we'll see the two different features collide, so maybe implement recent history ranges first
< wumpus>
or I mean first 20GB + last 144 blocks
< gmaxwell>
For history storage, I was previously working on a proposal where nodes could signal a small (32 bit) seed and a size and from that everyone would know what parts of the history they would store. I was so far unable to unify two different schemes, one which was computationally efficient to figure out who had what, and one which never required a peer to fetch a block it had previously deleted.
< sipa>
so very quick breakdown: out of 7M requested blocks, 100k were for the tip, range 2-2500 has around 200-2000 requests per block, and from 10000 to genesis deep there are around 20 per block
< gmaxwell>
I think for now we should not worry about the old history part and only worry about Top-n vs everything, as that fits into the pruning we already have and can be accomplished purely with service bits.
< wumpus>
the bittorrent problem is different in that there the goal of each node is to have everything
< petertodd>
so a social consideration here, is we can think in terms of recent history as "if there's a flaw, how much would we ever reorg w/o just saying bitcoin has failed?"
< gmaxwell>
petertodd: thats partly why we have the 288 block maximum amount of pruning.
< petertodd>
gmaxwell: indeed, and that's only two days...
< jonasschnelli>
Using multiple service bits for 4 ranges seems to be a hackish-design IMO
< gmaxwell>
at 100 blocks any reorg will _necessarily_ cause unrecoverable losses. So 288 basically gives a day plus an extra day for overhead.
< petertodd>
there's also a natural time criteria from how the difficulty adjustments reduce your resistance to 51% attack - if your node is offline longer, the minimum attacker size to fool you goes down
< sipa>
strangely enough: i see much more requests around 1000 deep than around 100 deep
< gmaxwell>
jonasschnelli: I don't see anything hackish.
< wumpus>
jonasschnelli: I also think it's a strange use of service bits
< jonasschnelli>
I'd prefere using a single service bit to state pruned blockchain and then a new message (or append something to version?)
< petertodd>
sipa: probably because people don't turn their nodes on and off every day
< gmaxwell>
sipa: you probably want to filter out the bitnodes spider, as I believe it requests a block to check the node is working.
< sipa>
gmaxwell: ah.
< gmaxwell>
petertodd: someone who hasn't turned their node on will request all of 0 to -1000. so it will not make 1000 greater.
< gmaxwell>
jonasschnelli: NAK.
< petertodd>
gmaxwell: oh! I didn't know we did that
< sipa>
i'm a bit surprised people think there is no need to have the available block ranges indicated in addr messages
< sipa>
(whether through service bits, or some extension)
< jonasschnelli>
I think there is a need... but it could be a second step
< wumpus>
jonasschnelli: appending to version should be unnecessary, that's also a hack :)
< sipa>
jonasschnelli: if it's a second step, we need to extend addr, and the whole management of addresses
< jonasschnelli>
Okay. Agree. What about a new message type?
< jonasschnelli>
blockrange
< sipa>
jonasschnelli: you don't understand.
< gmaxwell>
jonasschnelli: look at pieter's request figures, if nodes are effectively forced to go to peers that have everything whenever they connect becuase if they don't know they'll be able to fetch any blocks at all, then it will put lots more load on them.. causing people to stop offering blocks... causing more pressure on what remains.
< sipa>
jonasschnelli: the point of having it in service bits is so nodes can find peers that have the range they need
< wumpus>
but addr information gets old really fast
< sipa>
wumpus: much less so with feeler connections
< wumpus>
nodes may dynamically change what blocks they have, so there will always be cases of nodes connecting and realizing they have nothing to offer each other
< jonasschnelli>
Okay. I see the point.
< sipa>
(presumably, i don't have numbers)
< wumpus>
just like currently nodes will try to connect into black holes that no longer host a node
< petertodd>
so another interesting thing here is that ranges are queried linearly - you download blocks in a roughly linear fashion - so we could take advantage of that by making sure that nodes with one range keep track of nodes with adjacent ranges
< wumpus>
sipa: sure, feeler connections make it somewhat better
< gmaxwell>
wumpus: yes, sometimes the data is wrong. But there is a big difference between having 80% of the nodes on the network giving you no idea if they'll be useful at all until after you connect, vs a suggestion that might sometimes be wrong.
< wumpus>
but I don't think addr is a very up-to-date information source
< petertodd>
thus, as you sync the first time, ask nodes with the range you're syncing at this moment for the next range you need
< luke-jr>
wumpus: if ranges are deterministic, they don't need to be up to date
< sipa>
petertodd: yes, any sharding plan wouldn't randomly distribute the kept blocks, but keep randomly distributed ranges
< gmaxwell>
wumpus: I don't know if you realize that sipa and I are not thinking in terms of absolute ranges here. but nodes saying "I keep the last 288" or "I keep the last 2016" or "I have all of history".
< wumpus>
gmaxwell: but indeed this is a different problem from the bittorrent problem where everyone's goal is to have everything
< sipa>
gmaxwell: well that's sharding... maybe that is something to postpone for later
< petertodd>
sipa: sure, I'm more talking about how the linearity affects the network p2p design - prefentially peering with peers with the adjacent range may even be a reasonable design
< luke-jr>
wumpus: eh, everyone needs to get everything
< wumpus>
there, nodes can just connect randomly and have a high change the other nodes has something to offer them
< gmaxwell>
wumpus: and I wouldn't expect that data to go out of date fast.. pretty much only when nodes go up and down.
< sipa>
oh, nvm, i'm misreading
< wumpus>
luke-jr: only initially
< luke-jr>
oh, I see the distinction
< wumpus>
luke-jr: bittorrent nodes don't throw away blocks, generally
< luke-jr>
f(best-height, seed-in-addr) -> ranges
< gmaxwell>
for the spreading the history around, as mentioned I came up with concrete schemes (based on consistent hashes) that have nice properties.
< sipa>
i wonder whether we need to have that in the first go at this
< jonasschnelli>
I think a first simple solution that allow to extend it further would be appriciated.
< sipa>
even just having serve-everything and server-the-last-288-and-relay-at-tip would be a good addition
< wumpus>
making the ranges deterministic makes some sense, on the other hand, it does restrict the flexibilty of nodes to choose what ranges they host, it means everything has to be got right in first try
< gmaxwell>
sipa: thats what I am saying.
< jonasschnelli>
sipa: agree
< gmaxwell>
I do not think we can do better immediately anyways.
< sipa>
21:18:07 < jonasschnelli> I'd prefere using a single service bit to state pruned blockchain and then a new message (or append something to version?)
< gmaxwell>
sipa: though your latest figures suggest that the 2016 depth is important too.
< sipa>
21:19:07 < gmaxwell> jonasschnelli: NAK.
< petertodd>
if nodes attempt to maintain a few connections to peers that have the next range after they have, maybe it doesn't matter exactly what the ranges actually are? any given node would have a few connections to the next range, and anyone syncing from them could ask for those connections
< gmaxwell>
sipa: my understanding of jonasschnelli comment was there should be a bit that says "I relay blocks but don't have history" I am NAK on that.
< wumpus>
as there is no scope for later optimization, because all nodes have to agree what ranges are implied
< jonasschnelli>
We could add a service bit that says "I relay only the last 288 blocks"
< wumpus>
jannes: yes that would be the initial idea
< wumpus>
jonasschnelli*
< sipa>
gmaxwell: how is that different from what i suggested?
< sipa>
21:26:10 < sipa> even just having serve-everything and server-the-last-288-and-relay-at-tip would be a good addition
< jonasschnelli>
I think my initial idea with the general pruning sevice bit and a new message type is to complex and inflexible
< gmaxwell>
jonasschnelli: yes, that would be better, though pieter's data suggests that there are a LOT of requests at 1000. I think if I had that data I would have been suggesting the maximum pruning should be 2016, and then had the bit at that dep.
< gmaxwell>
sipa: the ability to relay blocks at depth -10.
< sipa>
gmaxwell: less than 2% of blocks requested from my node are at the tip
< sipa>
(but the tip is still 100x more frequent than any other individual depth)
< sipa>
gmaxwell: "a service bit to indicate pruned blockchain" implies you can serve 288 deep :)
< petertodd>
gmaxwell: re: maximum pruning depth, it's reasonable for that to be a similar % of the total data that storing the UTXO set takes - if you have 10GB of UTXO, 2GB of block data isn't a big change
< wumpus>
yes, you could define it as that
< gmaxwell>
I don't think there is any remaining disagreement on using bit(s) to signal I have a top-n. But I have some doubt on N. it needs to capture the largest amount of the block realy bandwidth without being unduely pruning incompatible.
< wumpus>
288 is the minimum pruning amount in bitcoin core already so it'd be a valid choice
< morcos>
as a first pass, i wonder if you preferentially downloaded from pruned peers whenever you were behind by less than 288 blocks, that would take enough load of peers serving full history?
< gmaxwell>
morcos: absolutely.
< jonasschnelli>
Good idea
< wumpus>
yes, that would make sense
< gmaxwell>
unfortunately, sipa's data suggests that 288 sheds less traffic than measurements years ago suggested.
< sipa>
maybe i should compute statistics in bytes rather than blocks
< morcos>
gmaxwell: it wasn't clear to me what the integral from 1 to 288 was compared to 288 to inf
< wumpus>
well it is a compromise
< wumpus>
putting the threshold higher makes some peers completely useless
< sipa>
to see what percentage of bandwidth is needed in 1-288
< wumpus>
which reduces morcos 's argument
< jonasschnelli>
Yes. I guess you convinced me to use two service bits then. -288 and -2016
< gmaxwell>
which is why it might be useful to use two bits and be able to signal 1-288, 1-2016... and perhaps start encouraging people to not prune shorter than 2016.
< sipa>
i think we're getting into a design discussion here
< sipa>
my number are very premature and not well analysed
< wumpus>
it'd also be possible to add a 288-flag now, and then consider a 2016 flag later
< gmaxwell>
sipa: indeed, thought that was the input you requested from me.
< morcos>
wumpus: yes, thats what i'm saying
< gmaxwell>
wumpus: yes! indeed.
< jonasschnelli>
Agree with wumpus
< wumpus>
if it turns out to be necessary
< petertodd>
wumpus: ACK
< sipa>
yes, i think just a 1-288 one seems useful
< wumpus>
good :)
< jonasschnelli>
Start with a simple tip-288 relay, and get some experience
< gmaxwell>
wumpus: it looks pretty clearly necessary but no need to do everything at once.
< petertodd>
wumpus: basically advice is, turn your node on at least once every two days
< wumpus>
petertodd: yes
< gmaxwell>
petertodd: we really should have cron mode for the daemon where it just syncs up and shuts off. :P
< gmaxwell>
bitcoind -oneshot
< gmaxwell>
:P
< petertodd>
gmaxwell: heh, that's not a crazy idea - I'd use it on my laptop
< jonasschnelli>
didn't we once had a proposal for the pause option?
< wumpus>
right, there's a flag that quits after reindex, but none that exits after sync
< wumpus>
would be easy to add tho
< morcos>
we could just ask for the utxo set, shoudl we discuss ideas how to do that
< CodeShark>
^ yes :)
< petertodd>
make -oneshot run in the foreground with a progress bar :)
< wumpus>
without utxo commitment that's a no-go
< morcos>
thanks codeshark
< petertodd>
wumpus: +1
< gmaxwell>
morcos: pointless when we were unable to get past the discussion for the security model change to not validate the past history based on proof of work.
< petertodd>
and lets not underestimate how dangerous UTXO commitments can be - I'm very dubious about committing to the (U)TXO set more recently than maybe a month or two
< CodeShark>
would be great to query utxo for quick sync, then go backwards in time fetching blocks to increase security...but yes, this is a design discussion
< morcos>
i was making a joke, sorry
< CodeShark>
alas, quick sync doesn't look feasible in the nearterm
< wumpus>
ok, next topic?
< gmaxwell>
but since that was brought up... Can we talk about removing checkpoints?
< wumpus>
#topic removing checkpoints
< sipa>
what % of transactions are before the last checkpoint
< sipa>
does anyone know?
< morcos>
someone should write up a design proposal for that to be evaluated
< gmaxwell>
Right now they're used for two things, preventing header flooding with low difficulty headers; and skipping signatures in earlier blocks.
< petertodd>
gmaxwell: just removing checkpoints, or assuming sigs are valid if buried deep enough?
< sipa>
gmaxwell: and 3) estimating progress
< wumpus>
keeping something for estimating progress would make sense
< sipa>
i think 1) remains needed and 3) remains useful
< wumpus>
that doesn't need to be checkpoints
< gmaxwell>
because very few percentage of the transactions are below the checkpoint .. since libsecp256k1 (and I expect the checkqueue)-- my point two is basically pointless, and I think it could just be removed
< gmaxwell>
I think on a desktop it only adds 15-20 minutes to the sync.
< petertodd>
gmaxwell: I'd ACK simply removing checkpoints entirely; I'm not happy to see them replaced with another scheme to skip sig checking
< wumpus>
a block-height-to-relative-difficulty map would have much less of a stigma
< wumpus>
eh, verification difficulty that is
< sipa>
gmaxwell: really?
< gmaxwell>
petertodd: I think we could remove CP from reason two without implementing the replcement.
< gmaxwell>
petertodd: morcos is right that needs a design proposal outside of the meeting.
< sdaftuar>
i'm a bit confused about how to think about checkpoints for signature skipping
< gmaxwell>
sipa: I benchmarked before but I'm going off of memory, I could be wildly wrong. I will test again if there is interest.
< jonasschnelli>
Removing checkpoints would slow down (maybe insignificant) a scan in a possible SPV hybrid mode?
< gmaxwell>
For reason (1) the only answer I have is that I think we should proposal a bit to perpetually increase the minimum difficulty from 1 to something else.
< sdaftuar>
for instance the recent ISM change caused us to do less validation for certain blocks in our history (blocks in a softfork between the 75% and 95% thresholds)
< sipa>
jonasschnelli: SPV mode won't validate *anything* at all
< gmaxwell>
(with a checkpoint like bypass of that new rule, for existing blocks that break it) As little as 100,000 would eliminate the header flooding vulenrablity.
< jonasschnelli>
Yes. But assume we would add an SPV hibrid mode in oder to received payment during IBD
< jonasschnelli>
One would need to download 400k headers without a checkpoint at h400k
< luke-jr>
maybe checkpoints should just be disabled by default before complete removal?
< sipa>
jonasschnelli: i think you're confused
< gmaxwell>
for Sipa's (3) reason for 'checkpoints' I don't give a darn, use chicken bones for progress estimation for all I care. :P it's historical accident that checkpoints and progress use the same data structure.
< morcos>
gmaxwell: :) +1
< wumpus>
gmaxwell: yes, my point too
< sipa>
gmaxwell: agree, those could be completely separated
< petertodd>
gmaxwell: ACK checken bones
< gmaxwell>
Might as well fit a cubic spline to the height vs txn count... and store the parameters.
< wumpus>
right
< petertodd>
gmaxwell: heh, if we do that with floating point math that has the advantage that it _can't_ be used for consensus :)
< * sipa>
now remembers a song our student organization wrote to the melody of staying alive, called 'cubic spline'
< gmaxwell>
so my proposal, if there is interest, is that I'll measure the performance impact of removing the signature skippingentirely (esp post checkqueue). And if it's not awful, we'll remove.
< wumpus>
+1
< sipa>
gmaxwell: i'm unconvinced
< wumpus>
it doesn't hurt to benchmark
< gmaxwell>
and maybe I'll tender a proposal to up the minimum difficulty, but I'd like to know what people think about that.
< wumpus>
measuring is always better than making assumptions
< sipa>
with a replacement for sig skipping that isn't based on checkpoints we could significantly improve things
< petertodd>
sipa: I don't think such a replacement can exist without changing the security assumptions; I'd *rather* have checkpoints than trusting hashing power for that
< sipa>
the last checkpoint currently is very old for the very reason that we've been planning to replace it
< gmaxwell>
sipa: would you like to help work on a proposal for that? it has been controversial in the past. I'd like to do something good, because otherwise imprudent attempts will be adopted instead.
< sipa>
so it's unfair to use the "the last checkpoint is old" as a given; it's something we've affected indirectly
< petertodd>
sipa: though what checkpoints should do is say "Something big has changed; you can disable checkpoints with --no-checkpoints, but you should find out what this means before doing so."
< gmaxwell>
(for example Bitcoin Classic's current behavior simply looks at block header timestamps and ignores signatures when they're more than 24 hours (*par) old by the local clock. It's easily exploited and makes me sad.
< sipa>
petertodd: it's my opinion that on a timescale of months, it doesn't matter
< sipa>
IF you can guarantee it's actually a timescale of months
< wumpus>
yes that makes me sad too
< petertodd>
sipa: on a timescale of months, checkpoints shouldn't matter either...
< wumpus>
anything based on time seems very brittle
< sipa>
petertodd: look at the current hashrate; what's 3 months worth of chain work at that hashrate
< petertodd>
wumpus: and anything based on work isn't much better if you're running an old client, and mining has advanced significantly
< jonasschnelli>
sipa: I (hope) I'm not confused. If we would add a SPV hybrid mode directly fetch blocks at the tip (in order to received payments), no available checkpoint would result in downloading all headers *losing* maybe 3-4mins before you can start using SPV... minor issue though, I agree
< petertodd>
sipa: that assumes you know what the current hashrate is
< gmaxwell>
wumpus: the prior proposals were based on work, e.g. skip if the best chain you see dominates the next conflicted chain at that hight by N months of work.
< petertodd>
sipa: your node might be surrounded by sybils
< gmaxwell>
wumpus: with a 'minimum total work' coded in as part of the release proces.
< sipa>
Chris_Stewart_5: headers first sync
< sipa>
Chris_Stewart_5: 0.10
< gmaxwell>
Chris_Stewart_5: headers first sync.
< wumpus>
gmaxwell: right. Well, at the least it should be measured whether such a change is really worth it.
< sipa>
petertodd: yes, i know...
< sipa>
so, let's measure.
< sipa>
and discuss later
< gmaxwell>
Chris_Stewart_5: and the signature skipping behavior in checkpoints was actually a result of a bug fixed years ago.. mlock being used on all allocations making script validation INSANELY slow.
< wumpus>
so much of the verification overhead is looking up UTXOs
< gmaxwell>
sipa: okay.
< wumpus>
something you'll not avoid
< gmaxwell>
Chris_Stewart_5: but then with chain growth we became dependant on it to keep sync times reasonable. but libsecp256k1 made signature validation >5x faster.
< wumpus>
especially for recent blocks
< wumpus>
if you do any benchmarking please look at the recent blocks, not the first N
< gmaxwell>
wumpus: it's still a major speed up on existing blocks.
< sipa>
in a side node: i've already updated my logging to measure bandwidth vs blockdepth instead of just count.
< Chris_Stewart_5>
So header sync solves the attack of flooding disk space, but not having your entire network hijacked, correct?
< wumpus>
Chris_Stewart_5: huh?
< wumpus>
gmaxwell: sure, could be
< gmaxwell>
Chris_Stewart_5: isolation can be resolved by simply knowing what the total work of the best chain was at release.
< gmaxwell>
Chris_Stewart_5: sorry, this was discussed prior times removing checkpoints had come up, I haven't completely described the background.
< Chris_Stewart_5>
gmaxwell: Thanks for the explanation, i'll keep digging.
< wumpus>
Chris_Stewart_5: ah, you mean being isolated and being fed a wrong chain, sorry I was imaginging some wacky things at having your network hijacked :)
< wumpus>
ok, next topic?
< gmaxwell>
wumpus: just the "you got a faithful bitcoin core download but the attacker controls your network"... but that doesn't need a checkpoint to fix, a simple partitioning detction that knows the total work of the best chain at releast time is sufficient.
< gmaxwell>
Thanks for the discussion.
< wumpus>
#topic segwit against uncompressed keys or not
< wumpus>
(10 minutes to go)
< wumpus>
(9 minutes to go)
< petertodd>
so to be clear, *just* segwit right?
< CodeShark>
does anyone still use uncompressed keys?
< wumpus>
yes, only segwit
< achow101>
CodeShark: armory does
< luke-jr>
seems uncontroversial
< petertodd>
I'm happy to ACK that given just segwit
< achow101>
having segwit enforce uncompressed keys would delay segwit adoption for armory users
< achow101>
*compressed
< jl2012_>
it's in #8499
< luke-jr>
achow101: why? just compress them
< wumpus>
gmaxwell: yes, though we had a lot of trouble with partitioning detection, I remember some code being stripped out and such. But anyhow, yes that's the better approach if it can be gotten to work.
< sipa>
achow101: sigh, does armory still not do that?
< achow101>
luke-jr: we have to change the whole wallet structure (it's still going to happen anyways)
< wumpus>
gmaxwell: without too much false positives
< luke-jr>
achow101: why?
< sipa>
achow101: alan said somewhere in 2013 he was implementing it...
< achow101>
alan's gone now..
< luke-jr>
afaik the only downside to using compressed keys is it changes the address, which segwit is changing anyway
< CodeShark>
it's not a very complicated change
< wumpus>
armory still uses uncompressed keys?!
< luke-jr>
there's no reason you'd need to change the wallet structure I can see
< wumpus>
in any case this only applies to segwit, not to old transactions
< achow101>
the plan is to have a new wallet structure with bip32 that supports segwit and compressed keys
< gmaxwell>
wumpus: "you're partitioned until you see a header chain with at least work X" is a pretty simple critera. :P
< sipa>
luke-jr: it had fixed size records in its wallet format for pubkeys
< sipa>
achow101: well if a new wallet format is needed for segwit anyway, it doesn't matter right?
< gmaxwell>
achow101: oh god please do not use uncompressed keys with segwit. why would you do that?
< luke-jr>
sipa: zero-pad it?
< achow101>
sipa: well no, we don't need a new wallet for segwit as it could still work with the old one with a little bit of hacking
< achow101>
that was the original plan
< luke-jr>
achow101: no less than compressed could
< luke-jr>
sipa: or store the uncompressed key, and compress it at address-generation/signing
< gmaxwell>
achow101: why cant the same hack that indicates segwit is in use indicate compressed.. you just chop off some bytes of the key pretty much.
< sipa>
btw, uncompressed keys account for 0.7% of used keys in succesful sigs on the network (in the past 2 hours)
< gmaxwell>
it could be done entirely inside the process that seralizes the segwit scriptpubkey.
< achow101>
gmaxwell: idk. ask goatpig
< gmaxwell>
achow101: okay
< * michagogo>
pokes his head in belatedly
< CodeShark>
I think we should encourage all wallets to use compressed keys - achow101, if you need help with this I'd be willing to help
< sipa>
agree - we should help
< gmaxwell>
yes, lots of people would be glad to help.
< sipa>
instead of just yell
< gmaxwell>
well I offered to help armory move off uncompressed keys to alan several times, including offering to pay to do it.
< gmaxwell>
so please don't say anyone just yelled.
< CodeShark>
I initially designed my account structures to only use compressed keys - but later added a compressed bit to support legacy stuff
< petertodd>
CodeShark: what legacy stuff specifically? legacy armory users?
< wumpus>
CodeShark: bah,it's kind of sad that to hear some things seem to be going back instead of forward :)
< CodeShark>
yes, to support other wallets
< wumpus>
it's time
< CodeShark>
but I think we really do need to prod all wallets to move to compressed keys
< CodeShark>
there's really no reason to continue to support uncompressed keys - other than perhaps some migration tools
< wumpus>
#endmeeting
< gmaxwell>
CodeShark: as pieter notes, virutally nothing is already.
< lightningbot>
Meeting ended Thu Sep 29 20:00:15 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< wumpus>
well supporting it in consensus for the normal network keeps making sense, but segwit is just such a great oppertunity to get rid of it
< petertodd>
wumpus: yeah, I don't see how we can remove backwards compatibility for it w/o confiscating funds, but no reason to not remove support in new addresses
< wumpus>
petertodd: indeed
< gmaxwell>
yes, thats why its important to get rid of now. otherwise I wouldn't care if action were taken n months later.
< luke-jr>
if anything, we should be discussing whether to make it a consensus rule rather than a policy ;)
< gmaxwell>
luke-jr: I like to but many people feel that addining an additional consensus rule for segwit now wouldn't be prudent.
< gmaxwell>
making it non-standard is sufficient in my view, such that we'd be able to make it a consensus rule later.
< btcdrak>
achow101 seems to be having connection problems
< luke-jr>
sure
< achow101_>
btcdrak: just a little bit. switching computers
< CodeShark>
gmaxwell: having to deal with the additional case complicates implementations
< luke-jr>
just saying the rule shouldn't be controversial itself really
< CodeShark>
not very much, but still
< petertodd>
gmaxwell: well, so long as we loudly warn that this is intended to become unspendable later if you bypass the standardness
< morcos>
gmaxwell: if so, we should be as clear about it being not allowed now as if we were to make it a consensus rule now.
< gmaxwell>
for those thinking that we have to verify all the old stuff for all time, that might be true for bitcoin core, but in the future I could imagine some implementations just not bothering to verify old stuff.
< gmaxwell>
morcos: petertodd: agreed for sure.
< sdaftuar>
petertodd: ntoe that it's standard to create an output with an uncompressed pubkey hash, as we can't detect the issue until a spend attempt (right?)
< jtimon>
I agree with luke-jr on not seeing the controversy in it being consensus rule with the rest of segwit
< petertodd>
gmaxwell: I don't mean verifying old stuff, I mean verifying new txs spending old coins
< petertodd>
sdaftuar: yes, nothing we can do about that though
< gmaxwell>
jtimon: well in the case of low-S which we also wanted to make a consensus rule, jl2012 discovered that there were corner conditions we wanted to think about more carefully before making it a consensus rule.
< sdaftuar>
petertodd: right; just want to make sure we're all on the same page as i think communicating this widely/loudly is important
< petertodd>
for example, we've made hybrid pubkeys non-standard, and given basically no-one's ever used them in production for anything I'd have no issues with making them unspendable in a soft-fork
< jtimon>
gmaxwell: thanks
< gmaxwell>
jtimon: I don't think the same applies to uncompressed keys, because the criteria there is even simpler. but the lowS reason is part of why we punted this collection of improvements to policy for now.
< jtimon>
mhmm
< michagogo>
I saw a movie that depicts a form of distributed/decentralized system, to avoid it getting shut down. Or in the words of the character that explains it, "everyone that logs on is a server". It's said to be "open source", but then that's explained as "anyone can edit the code, like Wikipedia.
< achow101_>
so if anyone wants to help armory with segwit support, bip32, compressed keys, we accept PRs. All our work happens in the dev branch, not master
< michagogo>
And the code is deployed when a majority of users approve it
< wumpus>
michagogo: heh, open source in some weird twisted mirror world
< gmaxwell>
achow101: is there a IRC channel where things are discussed? E.g. where should I ask goatpig about compressed pubkeys in segwit.
< sipa>
michagogo: i believe they're mistakingly not describing a computer network, but politics.
< achow101_>
#bitcoin-armory
< jtimon>
michagogo: that is, when sybil decides so...
< luke-jr>
achow101_: meh, should just collapse that into #bitcoin-dev :p
< michagogo>
And it's completely vulnerable to Sybil attacks…
< michagogo>
Gah, lagging
< michagogo>
Yeah
< michagogo>
And of course, when the last user logs off, it doesn't just stop working
< michagogo>
The sybil attackers are able to watch it dramatically implode with special effects, "graphical corruption" type stuff
< michagogo>
And there's the obligatory "they're blocking all our foreign IPs" and that kind of stuff, with no explanation of who "they" are
< jtimon>
so what was there a conclusion for the range service bits? nothing/top-288/everything?
< jtimon>
what about the getrange message and "sharding"
< GitHub159>
[bitcoin] laanwj opened pull request #8843: rpc: Handle `getinfo` client-side in bitcoin-cli w/ `-getinfo` (master...2016_09_getinfo_clientside) https://github.com/bitcoin/bitcoin/pull/8843
< wumpus>
conclusion was to add one service bit: last-288-served
< wumpus>
and maybe later one for last-1000-served
< jtimon>
wumpus: I see, and leave the rest for later, thanks
< luke-jr>
1024 would be rounder. ☺
< wumpus>
and a jackpot for whoever enabled both at once
< luke-jr>
if you set both, does it mean last 288000? :P
< jtimon>
would it be crazy to just have last-1024 without last-288 and just change prunning's default?
< wumpus>
288 is not just the default, it's the minimum
< wumpus>
I'd be okay with changing the default not the minimum, but that'd keep some nodes completely useless
< wumpus>
whereas by far most requests are in the last 288
< luke-jr>
wumpus: useless for syncing*
< luke-jr>
frankly, there are enough full-archive nodes out there that we really don't *need* to do anything right now, so meh :p
< sipa>
wumpus: actually, not true.
< jtimon>
well, the users know what to do to stop being useless...
< wumpus>
which, as morcos remarked, preferentially downloading the last blocks from would take a lot of load of nodes that do keep more blocks
< sipa>
there are more requests in 101-1000 deep then 2-100 deep
< wumpus>
ok...
< sipa>
*than
< wumpus>
I misremembered apparently, never mind
< luke-jr>
an unsyncable-from node is still more useful than a syncable node that isn't used for a wallet
< luke-jr>
syncable-from*
< jtimon>
maybe we can changethe prunning minimum if that simplifies things?
< sipa>
wumpus: well, sample of 1 long-term-running node over the course of a few weeks of data
< sipa>
wumpus: more samples welcome
< wumpus>
sipa: do you have a special patch for statistics collection?
< gmaxwell>
sipa: need to filter out bitnotes.
< sipa>
gmaxwell: right; how do you suggest to do that?
< wumpus>
sipa: or a script for parsing logs?
< sipa>
wumpus: both; i'll publish them after a little cleanup
< wumpus>
I could put it up on a few nodes, no problem
< sipa>
it just logs an extra line with depth and block size for each requested block
< wumpus>
nice
< jtimon>
I guess it's not completely crazy, but nobody seem to specially like it
< sipa>
en then
< sipa>
S=0; fgrep DEEP ~/.bitcoin/debug.log | cut -d ' ' -f 4 | sort -g | uniq -c | tac | while read C D; do S=$(($S+$C)); echo "$D $C $S"; done | tac | less
< sipa>
to inspect :)
< wumpus>
jtimon: no, it's not completely crazy, using only one service bit is kind of elegant
< jtimon>
:)
< wumpus>
jtimon: if 1000 is really one-size-fits-all, and <1000-keeping nodes may as well be ignored. It's just hard to say without better statistics.
< wumpus>
also statistics about what pruning sizes people prefer
< wumpus>
I mean if everyone prefers the minimum and no one sets 1000 in practice
< gmaxwell>
sipa: just do Satoshi:(recent) useragents.
< jtimon>
well, independenlty of the statistics we will eventually need a more generic solution for flexible sharding, right?
< sipa>
jtimon: maybe
< sipa>
"need" is a big word imho
< sipa>
but i agree it would be nice
< gmaxwell>
jtimon: I think we do, would you like to finish the solution for that I started on?
< wumpus>
jtimon: well there needs to be a different solution for historical block hosting IMO, but that's a different thing
< gmaxwell>
sipa: I think excepting participants to keep around hundreds of gigs of blockchain is not conducive to the surival of the network, the alternative I see is a hardfork that drops off the history past some point. (e.g. just restarts the chain from a utxo commitment made a year before)
< sipa>
gmaxwell: well, or just stop supporting historical block fetching more than 1 year or whatever number back on the p2p protocol, and use http
< wumpus>
or bittorrent *ducks*
< jtimon>
wumpus: yeah, historical hosting is what I mean
< jtimon>
gmaxwell: maybe, but it sounded deterministic like luke-jr proposed instead of flexible like wumpus wanted
< wumpus>
it could be anything that supports downloading ranges of data...
< brainwave>
Under overview, balances, on the right side of available, pending, total, add ~ exchange rate for dollars, pounds, euro
< sipa>
brainwave: bitcoin core does not and cannot know exchange rates
< sipa>
(because it would require contacting a centralized service, which we don't do by design)
< wumpus>
yes or someone would need to commit them to the chain, but that'd still be trusting a central issuer/signer of the information
< wumpus>
it's just a no-go
< gmaxwell>
well if the users of bitcoin accepted that kind of security model change, what I would suggest is something like every 26280 blocks the block is required to have a commitment to the utxo set (could be a linear hash) as of 2016 blocks prior. and then six months of work after that, that commitment becomes usable for initial sync. and so then no one need process more than a year of blocks at sync.
< gmaxwell>
.. though you would have to store three copies of the utxo set (though perhaps deduplicated)
< gmaxwell>
jtimon: I don't know why anyone would find determinstic less desirable.
< sipa>
gmaxwell: well i expect the controversy to not be about the change in security model, but about the perpetual requirement of having a utxo set
< wumpus>
gmaxwell: I explained that: if you make it deterministic you have to be sure of the parameters in advance, there is no room for tweaking or optimizing later on
< gmaxwell>
wumpus: well you simply extend the protocol to have a new signaling mechenism for the tweaked thing.
< wumpus>
sipa: yes the bigger problem is the ever-growing UTXO set
< wumpus>
gmaxwell: but then it loses backwards compatibility every time
< gmaxwell>
something that just signals absolute heights has the problem that the communicated information will always be out of date. .. or if nodes don't change the ranges they host, we will end up with highly irregular distributions of information.
< sipa>
the type of tweaking needed, and the potentially aging problem depend on the specific proposal
< sipa>
i'm sure we can come up with something that seems reasonable to all
< wumpus>
agree, there may be a compromise that is somewhat flexible and still deterministic
< gmaxwell>
well what I suggested might not be viable after all too. I'm not sure, I wasn't successful in achieving all my goals at once.
< wumpus>
I just don't think setting it all in stone in advance is a good idea, for the whole reason that it's so hard to achieve all your goals all at once
< wumpus>
especially if you don't know some of those goals yet
< gmaxwell>
I wanted a scheme that would result in a uniform distribution of blocks, that didn't depend on peers to look to see what other peers had (because that could be spoofed), required minimal communication (not a long list of blocks in an addr message).. and retainined uniformity as the chain grew, without causing peers to redownload blocks they already forgot.
< gmaxwell>
So I had found two schemes, one where peers had a ID and the amount of blocks they would store, and from that they could determine which they would store, and as new blocks came in they might store then and drop some group of old ones. The problem with it was that to figure out if a particular peer had block X you had to do computation linear in the number of blocks in the chain.
< wumpus>
darn, also fingerprinting will be hard to avoid
< gmaxwell>
Then I had another scheme that was sublinerar work, BUT a peer might drop a block but later have to go fetch it again.
< gmaxwell>
wumpus: thats unavoidable with any split up scheme.
< wumpus>
yes
< sipa>
make the IP address part of the seed
< sipa>
if your dhcp changes, you have to resync, sorry.
< wumpus>
unless a substantial part of nodes are the same
< gmaxwell>
sipa: then when you change IP, you have to go download a different set of blocks.. :P hah
< wumpus>
e.g. there are only 8 IDs, pick one
< gmaxwell>
wumpus: well I was thinking 32 bits, but perhaps a smaller collection would be fine.
< gmaxwell>
but that gives you at best only 1/8th spitting storage. :( maybe fine now, but not in the long term.
< wumpus>
maybe the number of groups can grow over time, a doubling every so many blocks :)
< sipa>
hah: if you get a request through an IP that doesn't correspond to your local storage, just proxy all requests through to another node which does, and use that to gradually resync for the new seed.
< gmaxwell>
Part of why I haven't given this that much more thought is because I think bitcoin will need to move to the commit state and forget history model; the ever growing sync time is too big a tide to stand against.
< gmaxwell>
sipa: lol!
< gmaxwell>
sipa: I think thats actually how the freenet location swapping works, funny enough.
< wumpus>
hehe
< sipa>
downside: if you want this to be fingerprint resistant, you have no way to determine how many proxies your blocks actually went through
< sipa>
=> instant mixnet
< gmaxwell>
sipa: freenet nodes change position over time, and they do it by swapping their location with a direct neighbor, when that location swap makes them both closer to where they want to be, ... when requests come in for the new location, they don't have the data, but it's only one hop away..
< wumpus>
gmaxwell: I've always thought that, it's hard to imagine this continuing for 10's of years, but where to put the anchor...
< gmaxwell>
in any case. if there were only 8 flavors of nodes, then it all becomes simple, block_height//1000 % 8 = flavor.
< * gmaxwell>
lunch
< wumpus>
that seems kind of elegant and straightforward, there must be a catch
< jtimon>
gmaxwell: sorry, well the deterministic seems to come at the cost of less flexibility
< sipa>
wumpus: i'm trying to think about why 8 isn't enough
< wumpus>
if you want to automatically scale the number of flavors up with height you could divide height 0..N into X flavors, the N..3N into 2*N flavors, and so on, where each flavor gets flavor (x<<1)+randbit()
< jtimon>
only 8 flavors requires you to store 1/8 of the blockchain
< sipa>
and we could have names for the first 8 top-level flavours or so... so your wallet could report "Looking for a bittersweet node..."
< wumpus>
(well those numbers are arbitrary but the idea is that if a doubling of the # is needed, the new flavor, a member of a twice as big set, would contain the previous one)
< GitHub72>
[bitcoin] jnewbery opened pull request #8845: Don't return the address of a P2SH of a P2SH (master...trivial-P2SH-P2SH) https://github.com/bitcoin/bitcoin/pull/8845
< gmaxwell>
jtimon: "less flexible" -- everything is less flexible short of sending someone arbritary x86 bytecode that they run.
< jtimon>
less flexible in the amount of data you store, but maybe 8 flavors can be subidivided in 16 flavors half the size as wumpus was suggeting, then 16 to 32, etc. That may be flexible enough
< gmaxwell>
jtimon: I was recommending 2^32 'flavors' but wumpus was concerned about reducing fingerprinting.
< gmaxwell>
the whole reason to reduce the amount was to make it more difficult to follow a node around as it changes network identity.
< gmaxwell>
sipa: 8 isn't enough if the chain is perpetually growing.
< jtimon>
I see
< sipa>
yeah, increasing number, the further back you go, may make sense
< gmaxwell>
a year from now the chain will be 200 gb, a year after 300 gb-- at that size it is larger than the most common ssd size currently. a year after that 400gb.... and at that point an 8 way split is again running common hosts out of disk even if the common ssd size has moved up to 500gb by then.
< jtimon>
well, maybe archive nodes that don't want to store everything have to get a privacy hit
< gmaxwell>
who will bother running one if it takes speical effort above and beyond running a node, and draws more resources?
< sipa>
well if only we'd have a separate network for archivsl
< sipa>
there are no privacy issues at all then
< gmaxwell>
and no one run them.
< gmaxwell>
s/run/running/
< sipa>
i was about to say that separate network doesn't need to imply separate nodes
< sipa>
but of course, that doesn't work because you'd get a privacy leak from correlating
< sipa>
however, you can reconcile those by only having nodes with a long-term IP provide archival further back than some threshold
< gmaxwell>
sipa: not just that, but if it's a special very resource intensive mode.. few will do it, pliling more resources onto it... causing fewer to do it...
< sipa>
it's true that it's resource intensive, but it's a different kind of resources than most of the rest of running a node
< sipa>
it needs disk space and bandwidth
< gmaxwell>
I might think it's not over the threshold of that, except already people don't run regular nodes due to costs.
< sipa>
rather than memory and cpu
< gmaxwell>
which are what people usually complain about.
< sipa>
then why aren't we seeing more pruned nodes?
< sipa>
one reason may be that pruned nodes don't advertize, so we just don't know about them
< gmaxwell>
because you have to edit a config file or change an obscure setting, we don't advertise it, and it breaks rescan and reindex. (which is part of why we don't really advertise it)
< sipa>
well people mostly complain about the sync time for a node
< gmaxwell>
yes, though I think thats most because so many stop there and give up before they get a chance to complain about the rest.
< sipa>
perhaps
< TD-Linux>
a first-run dialog box with a slider for disk usage and an estimated sync time would be very nice
< sipa>
except the sync time does not depend on the value of the slider
< TD-Linux>
yes, I meant it there so it'd appear at start. I guess having it in the status bar is sufficient
< sipa>
ah
< sipa>
well there will be an overlay with sync time indication in 0.14
< gmaxwell>
doesn't it still incorrectly say you can't transact while syncing?
< sipa>
we still have a lot time until 0.14
< gmaxwell>
:)
< wumpus>
well you can, but most people probably shouldn't do so
< gmaxwell>
yes they should
< wumpus>
during the initial sync they won't have any coins to send anyway, and receiving them is a bad idea as they'll only see them when the entire thing is done
< wumpus>
oh?
< wumpus>
why?
< gmaxwell>
initial sync isn't my concern there:
< gmaxwell>
probably one of the most common usage patterns for a wallet user is that you start your wallet up in order to pay someone, and it's three weeks behind. You can go ahead and pay, no problems.. why wouldn't you?
< gmaxwell>
during initial sync you just won't have any coins, indeed. :)
< wumpus>
the biggest problem is people giving out addresses during initial sync
< wumpus>
then realizing how long it takes
< wumpus>
this is what the overlay is designed to prevent
< wumpus>
sure, you can send coins if you're three weeks behind, no problem, although fee computation could be off
< gmaxwell>
yes, that is a large source of complaints, but we shouldn't tell people that they cant send funds already in their wallet when they start up and they're a bit behind, it's already a common mistaken belief that they cant (and then they complain about how long it takes to catch up a month of blcks)
< TD-Linux>
the warning could be conditional on having zero funds
< gmaxwell>
TD-Linux: the earlier warning text was fine-- saying that you won't see payments to you yet, but for some reason it was changed to say that you cannot send funds.
< wumpus>
yeah fix one thing and they'll start complaining about another, it's a never ending source of fun...
< sipa>
i don't think anyone will read the text anyway
< gmaxwell>
I also complained that the text is now too long and won't get read.
< wumpus>
of course people will read it
< sipa>
the important thing is that it's in the way, and gives accurate (by then, hopefully) information
< wumpus>
heck, users aren't stupid
< gmaxwell>
The first text was better.
< sipa>
gmaxwell: PR welcome
< wumpus>
maybe some are, but not all of them, some will actually read and understand
< gmaxwell>
Well I'm stupid, and looked at the notice in its updated state and didn't read the list line.
< gmaxwell>
first*
< gmaxwell>
because when there is too much text many people go a bit banner blind and skim past headings and such.
< wumpus>
if we don't believe peopel actually pay attention then why do anything at all
< gmaxwell>
saying that a wall of text is too much is not saying that people don't pay attention.
< wumpus>
I think it's an improvement to what was there, indeed, if you want to imrpvoe further then pulls are welcome
< sipa>
right, that's what i'm saying - having there being an overlay at all is more important than what the text says
< gmaxwell>
and re: being able to send, people already complain that they have to wait a long time after starting to send because they already frequently mistakingly believe they can't.
< sipa>
and we have time to improve the latter
< wumpus>
but I'm a bit tired of people always saying "users won't read anyway" to everything that adds documentation , help or warnings
< wumpus>
a lot of users are definitely looking for more help and guidance when they first open the program, and a bit of text helps there
< gmaxwell>
wumpus: why should I waste my time when I point out that THE TEXT IS OUTRIGHT UNTRUE and your response is to accuse me of thinking users are stupid? my comment was that the earlier version of the text which was simple and NOT UNTRUE was better.
< sipa>
please guys
< sipa>
gmaxwell: go propose something
< gmaxwell>
I did!
< wumpus>
gmaxwell: well if the text is wrong then it should be fixed obviously, change it to a better text
< wumpus>
I don't know what the previous version of the text was
< sipa>
it's been changed a dozen times in the lifetime of the pull
< sipa>
also, it says "Spending bitcoins may not be possible until synchronization has finished."
< sipa>
which is not untrue.
< gmaxwell>
okay, it was changed after I last saw it.
< wumpus>
ok that was useless :)
< gmaxwell>
by saying 'may' which is still misleading, but worse, that text is the bold.
< gmaxwell>
er is the only bold part.
< sipa>
well, improvements welcome
< gmaxwell>
So now it says "mumble mumble mumble Spending bitcoins may not be possible during that phase!" :-/
< gmaxwell>
it's a waste of my time, I already raised these issues and it was then merged.
< wumpus>
it had to be merged at some point, with the idea it could be improved later
< gmaxwell>
well to be fair the last change did improve it, its true.
< gmaxwell>
but created the problem that if you skim it is that all you extract is that you can't spend, .. which misses the really critical thing: which is that you wallet may look empty when it isn't.
< wumpus>
that doesn't mean it's final, most will only see the message when it is merged, and can improve it then, there are already some pulls open to improve that overlay
< gmaxwell>
but okay, I can open a PR.
< wumpus>
(but I don't think they change that message)
< sipa>
gmaxwell: i think people didn't really understand the point of your concern (i didn't): if you're looking at it from a point of view that this would be be mostly seen (and intended to convey information) during IBD, it's perfectly reasonable to warn users they won't be able to spend the money they're still to receive... and a simplification to reduce the length of the text may be warranted
< sipa>
it's a good point that it's also seen during non-IBD
< gmaxwell>
It will mostly not be seen during IBD.
< gmaxwell>
during IBD sure someone will see it then, say of course I knew that (even if they didn't) minimize and go on with life. :P
< gmaxwell>
but then users will see it every single time they start.
< sipa>
i'm aware, you don't need to argue about this
< sipa>
i'm just explaining why maybe you felt misunderstood
< gmaxwell>
sorry, not arguing-- clarifying.
< gmaxwell>
Yes, I see that and I didn't before.
< gmaxwell>
when I first saw this PR I even took the time to go through the code carefully to check to see if there was anything that made it IBD only.
< gmaxwell>
because I couldn't understand why people wanted the text that it had.
< gmaxwell>
it did not occure to me that other people might be only thinking about IBD.
< gmaxwell>
sorry for being thoughtless there.
< wumpus>
#8805 fixed a few minor grammar nits, #8821 fixes a blocking problem with the overlay, there are no pulls yet that improve the message
< sipa>
sorry, we (including me) aren't being careful with terminology here... IBD is also used for syncup when you were previously synced to a month ago
< wumpus>
it's very easy to forget about catching up nodes
< wumpus>
but yes we shouldn't
< sipa>
well it's mostly designed to help with that first sync
< gmaxwell>
more obvious to me just by chance of hering more people complain about it, also I've stopped running a node 24/7 on my laptop because I've been watching the battlestar galactica series in evenings and bitcoin interupts video playback. :)
< wumpus>
during catch-up it's reasonably useful too, people may not know they won't see transactions newer than their sync point and worry, but yes it's mostly important for the initial IBD (lol)
< gmaxwell>
so every time I go to use bitcoin I'm stuck waiting for it to catch up.
< gmaxwell>
yes, we should have this message during catch up. But it's important to not make people think they can't spend funds that they can see.
< gmaxwell>
The important message is that you may not see all payments to you yet (and you can't spend what you can't see).
< wumpus>
bitcoin interrupts video playback? even in steady state mode?
< TD-Linux>
one option would be to put it on the payment request generation page instead. but even in its current state it's far better than what was there before (nothing)
< gmaxwell>
wumpus: yes. on my laptop... playback from a local file. The issue is IO or cpu related, probably the former but I haven't tested extensively to know for sure.
< wumpus>
that's very strange. I'd expect that during intial sync when it maxes out CPU and I/O usage, but not when it's up to date
< gmaxwell>
TD-Linux: the big problem we should be solving here is that people see a balance of zero then delete the wallet. I think thats the priority because any other issue doesn't cause irrecoverable loss.
< gmaxwell>
wumpus: I notice it during ordinary computer use.. causes IO hangs, but its not irritating except when watching video.
< wumpus>
do you have a lot of mlocked memory? is it swapping?
< gmaxwell>
no, not swapping 8gb ram. I think that when a bunch of random writes happen it causes long delays for garbage collection in the SSD.
< wumpus>
swapping seems to be the foremost cause of I/O related hangs here, as essentially the memory subsystem has to wait for I/O to complete
< wumpus>
heh as if 8gb ram means no swapping these days :)
< gmaxwell>
well on my laptop its enough most of the time.
< gmaxwell>
The stalls seemed to get better for a while after I freed up a bunch of space and trimmed the drive, but got worse after which is why I think SSD GC plays a roll.
< gmaxwell>
but in any case, while watching the show every block arrival causes a second-long pause in playback.
< wumpus>
maybe someone is requesting a lot of blocks from you with a bloom filter? :-) it would be interesting to find out what your node is actually doing at those times
< gmaxwell>
nah, outbound only.
< gmaxwell>
I know its at the same time as blocks showing up.
< TD-Linux>
gmaxwell, easy way to verify that would be to increase your video player's lookahead cache
< wumpus>
ok so it's block verification, leveldb seeks
< gmaxwell>
TD-Linux: think mpv uses non-blocking reads of the disk?
< TD-Linux>
gmaxwell, yup it does. I've increase the setting to 10s when using sshfs and it works fine
< gmaxwell>
in any case, performance distraction aside, when this happens I shut down bitcoind then it may stay off for a week before I need to do something with it, then waiting for it to catch up is irritating.
< gmaxwell>
(and of course my systems performance is seriously impacted while it catches up)
< wumpus>
yes, nothing to do about that, I guess if hybrid SPV mode is implemented it could also work during catch-up
< wumpus>
indeed, it's either slow down the catch up or tolerate it hogging the whole system
< gmaxwell>
I have wondered if it might be useful to split the chainstate into two parts, one with txouts created in the most recent N blocks, and one with the rest. Then on start we could just load the whole first one into the cache.
< wumpus>
the default setting of hogging all cores during IBD/catch-up is a bit rude, certainly if it is a background process
< gmaxwell>
if we did that much of the cost would then be signature valdation instead of random IO, and signature validation could run in the background, following behind the blocks.. and at lower priority.
< wumpus>
so that would be like a 'prefer keeping recent UTXOs' cache policy?
< gmaxwell>
I guess thats a stat that I still haven't collected. "what is the average age-in-blocks of inputs that are consumed" (/what is the distribution of that age)
< gmaxwell>
we know recent ones are spent more often but I don't have good numbers on it.
< gmaxwell>
wumpus: yes.
< gmaxwell>
probably not the highest priority improvement in any case.
< wumpus>
well, currently the whole cache is emptied at a write, I think there are many eviction policies that would do better
< gmaxwell>
right, also the in memory representation of the cache entries is quite inefficient.
< gmaxwell>
so its effective size could potentially be doubled if its entries were flat allocated.
< wumpus>
though that helps it actually being an efficient cache, a more efficient representation shouldn't come at a higher access cost
< wumpus>
although that's for the entire UTXO set, not just a limited cache
< gmaxwell>
yea, the on disk seralization is most efficient, but not fast.
< gmaxwell>
I wouldn't be surprised though for the current in memory representation if there wasn't more bytes spend in malloc/container overhead than actual transaction data though.
< wumpus>
but given that most time is wasted on disk seeks anyway, it may not make too much difference in practice
< wumpus>
depends on the system...
< wumpus>
yes, the malloc overhead is somewhat bad
< wumpus>
but not more than the actual data size, from what I remember
< wumpus>
in any case improvements are certainly possible there, without any rocket science, it's just that it's such risky code to change
< wumpus>
if it was any other project people would have optimized the shit out of it by now
< wumpus>
unfortunately the damages of a bug there are unfathomable, not just skipping a few video frames
< gmaxwell>
well the hope I had there was that with making the cache more efficient, it could be increased in size and avoid more disk IO. :)
< gmaxwell>
on the earlier subject of 'alt' implementations doing inadvisable things, some of them have a genius performance optimization which they're crowing about, -- where they only validate transactions that weren't already in the mempool; something we explicity decided not to do because of the long history of subtle mempool corruption issues.
< wumpus>
exactly, there are tons of ways to optimize and get things just slightly wrong
< gmaxwell>
I worry that there will be a race to the bottom, where by making risky / security reducing optimizations implementations will gain significant performance advantages, and suffer no cost until the inevitable spectacular failure that results.
< wumpus>
which doesn't matter if no one runs your code anyway, but we have to be really careful
< gmaxwell>
and being safe doesn't matter if peopel don't run it in favor of things that are faster.
< gmaxwell>
people*
< wumpus>
we also shouldn't overestimate how important the performance is to most users, many just run it on a server or otherwise unused computer
< wumpus>
well you'd say safeness is really important, if the inevitable spectacular failure happens you don't want to be at the center of it
< gmaxwell>
well sure. indeed.
< wumpus>
better slow than dead :)
< gmaxwell>
but for things like the security model change to use a 6 month old utxo commitment instead of syncing the history... the potential for a spectacular failure there which a more conservative approach could have stopped is negligible.
< TD-Linux>
gmaxwell, still think you should verify that it's actually IO that's the problem before going too deep :)
< gmaxwell>
And if we don't investgate things like that, someone will do something dumber.
< wumpus>
TD-Linux: yes, measuring is better than assumptions :)
< gmaxwell>
TD-Linux: Oh IO is an issue for sure regardless of whats causing my mpv stalls. I'll find out tonight (I'm not going to sit here and watch video for an hour just to check)
< gmaxwell>
TD-Linux: SSD vs a fast spinning disk with small dbcache here is a <4 hour sync (from a local peer) vs a >9hour sync.
< wumpus>
and sure, it's good to investigate things like that
< wumpus>
I/O is absolutely a problem sometimes, leveldb generates *tons* of seeks and small reads, better caching would help avoid some of that
< wumpus>
I had no luck with other databases, I remember trying with lmdb at some point, which was faster in reading, but instead... does tons of seeks and small writes at write, so it just moves the problem
< TD-Linux>
yeah, on a SSD at least, the former is much less detrimental to system latency
< wumpus>
indeed!
< gmaxwell>
for operation at the tip, using the mempool instead of validating would be a big aid... but the safty of that remains dubious. :)
< sipa>
gmaxwell: i believe for the mempool it's approximately a factor 2 overhead
< sipa>
gmaxwell: that that is also indexes, orderings, accounting, ...
< wumpus>
yes that remains dubious, especially with regard to isFinal and such
< wumpus>
I mean there is a part of transaction validation that can be obviously cached, and a part that may change in time
< gmaxwell>
fortunately the MTP change made that much safer.
< gmaxwell>
e.g. before a block could come in with a time before your local time, and contain txn which are isfinal invalid according to the block but okay with respect to the local time, and you'd accept it. I wonder if the alt implementations had that bug.
< wumpus>
I remember that one, tricky... and there may be more problems of that kind not yet found
< gmaxwell>
looks like their change dodged it, because the finality test is in the ContextualCheckBlock, and the bypass patch only bypasses checkinputs...
< gmaxwell>
(which also means that it doesn't manage to avoid accessing the utxo cache entries)
< wumpus>
I just realized, if the problem is that the block validation hiccups other things happening on your PC, the solution may be actually to slow it down :)
< wumpus>
put a small sleep between each UTXO lookup, limit the validation to one thread
< wumpus>
not something you'd want to do during initial sync if you're waiting for it, but if you don't care and it runs in the background...
< wumpus>
after all you run it to keep up, you don't need to outrace it
< gmaxwell>
I've thought before that if we have bandwidth limiting enabled we should delay announcement of new blocks to reduce the number of peers that request them from us... but slowing down the validation would work as well.
< gmaxwell>
small sleeps perhaps aren't so good because it may busy spin. :P
< wumpus>
heh, not that small
< wumpus>
or use some OS-dependent way to reduce the I/O priority
< wumpus>
as long as it's done by the time the next block comes in, so taking 10 minutes would take it too far :)