< S3RK>
question about fuzzing. we use regtest chain params, but in corups for src/test/fuzz/descriptor_parse.cpp I see xprv keys. As a result fuzzing don't cover code paths with valid keys for bip32 decriptors. I tried to add initial seeds with tprv, but fuzzer doesn't detect any new edges covered. What do I miss?
< wumpus>
next general meeting we should go over the list
< jnewbery>
ok, onto the topics
< jnewbery>
#topic addrman file versioning
< vasild>
jnewbery: how did you suddenly come to the realization that old versions are not guaranteed to fail the parsing, out of the blue?
< jnewbery>
#19954 meant that in future it won't be possible to do forwards-compatible changes to the peers.dat format. #20284 is suggested as a way to allow that again.
< vasild>
pr20284 makes it possible to say in peers.dat "this is format=5, but anybody who can read format=3 can also read this one"
< luke-jr>
maybe the filename should just be changed if that's not the case, in the future?
< jnewbery>
vasild: yes, that's the functionality we want before the 0.21 release. Just want to make sure people are happy with repurposing nKeySize for versioning.
< sdaftuar>
jnewbery: sorry if this is a digression, but i might be misunderstanding the issue in #16702 -- it looks like the version check there just causes potential rebucketing, not losing addresses?
< gribble>
https://github.com/bitcoin/bitcoin/issues/16702 | p2p: supplying and using asmap to improve IP bucketing in addrman by naumenkogs · Pull Request #16702 · bitcoin/bitcoin · GitHub
< wumpus>
I think it's a hack but also a clever way of doing so, and he also introduces a real versioning mechanism for next time, so it's only for once
< sdaftuar>
so the issue that does appear to be there is that downgrading after running 0.20 may be a problem
< jnewbery>
sdaftuar: it means that entries in new tables can only appear in one new table (or even zero if there's a collision)
< wumpus>
definitely don't have any alternative proposals
< jnewbery>
the point of the serialization format is that entries can appear in multiple new tables
< vasild>
luke-jr: right, filename change is another option
< sdaftuar>
wumpus: we could rename the peers.dat file for 0.21, and migrate data from the old file to the new one?
< wumpus>
sdaftuar: that's a good suggestion too
< sdaftuar>
jnewbery: hmm i need to read that more carefully then.
< jnewbery>
sdaftuar: that would definitely prevent being able to keep your addrman data over a downgrade
< wumpus>
though it'd require updating a lot of documentation all over the place describing bitcoin core's files
< wumpus>
everyone is used to "peers.dat"
< luke-jr>
how often does anyone mess with peers.dat directly?
< wumpus>
I have no idea
< vasild>
btw, I did consider the file name change, but preferred the versioning inside the file, instead of using the filesystem for versioning
< wumpus>
yes, I slightly prefer this as well
< sdaftuar>
i think it's okay if once in a while we have to give users annoying instructions for downgrading.
< luke-jr>
IMO it depends on compatibility
< luke-jr>
if downgrading doesn't lose the address book, same filename is good; otherwise, a new name avoids losing the old addrman for the old ver
< jnewbery>
what would the annoying instructions be?
< sdaftuar>
we could add a tool that either converts formats, or an RPC or something
< luke-jr>
^
< wumpus>
sdaftuar: as I understand older versions will just ignore the incompatible peers.dat, and create a new empty one. But yes I agree. Downgrading is quite a rare event anyhow.
< luke-jr>
ew
< sdaftuar>
what did we do the last time we changed block file name conventions?
< wumpus>
sdaftuar: no one is going to go through that much work for their peers table
< luke-jr>
sdaftuar: iirc hard linked
< sdaftuar>
luke-jr: that doesn't sound backwards compatible?
< luke-jr>
why not?
< jonatack>
I must have lost my peers.dat a couple dozen times while testing the addrv2 PRs. within the next year, anyone still using tor v2 will be forced to upgrade to 0.21 anyway.
< sdaftuar>
oh we hardlinked new and old filenames together, i see
< wumpus>
I'm not sure it makes sense to extensively describe alternatives here, we have vasild's work, unless someone really strongly disagrees with it let's review it and get it merged asap
< jonatack>
unless they don't use tor
< jnewbery>
I don't think there's any need to change file names. It should be perfectly possible to have internal versioning for changes that are forwards compatible and not forwards compatible. I think that's what 20284 does, but I haven't reviewed it yet.
< wumpus>
assuming this really needs to go in for 0.21.0-otherwise there's no hurry
< aj>
afaics, for forwards compatible changes, we just don't bump the version number?
< sipa>
argh, timezones
< vasild>
I started a 0.20.1 node on my new peers.dat and it parsed it without emitting errors, so it is not just "theoretical"
< vasild>
surely it parsed everything as garbage
< wumpus>
it's definitely not theoretical
< vasild>
I don't know why during testing of that change, some weeks ago I always got some read/parse error, which fooled me that old versions fail to parse always
< jonatack>
vasild: same
< vasild>
:-D
< jnewbery>
aj: I think there needs to be versioning to show that there are semantic changes, even if they are forwards compatible
< wumpus>
garbage addresses would be kind of nasty, especially if they might also be gossiped and thus pollute the rest of the network too
< luke-jr>
jnewbery: point being if it isn't forwards compatible, the rename keeps the old file for the old version if you downgrade
< jnewbery>
luke-jr: that's a good point
< jnewbery>
So the action is to review 20284, unless anyone has anything important to add
< jnewbery>
#topic I2P support, some background at vasild/bitcoin/wiki/I2P-connectivity (vasild)
< jonatack>
When you lose your peers.dat, a new one doesn't seem to take very long to get up to a decent size (40-50k peers)
< jnewbery>
jonatack: it probably takes longer for your tried tables to get populated
< sdaftuar>
jonatack: the whole network losing their peers.dat in the event of downgarde might be a scary idea though (imagine a bug in 0.21, say)
< sipa>
jonatack: that may be deceptive though; just because you have a lot of rumoured IP addresses, they aren't necessarily good
< jonatack>
good points, thanks
< aj>
jnewbery: if you bump the version from 5 to 6 (eg), and 6 is "unknown" it will be treated as non-supported, so won't be forward compatible. you need major/minor versions if you want to do that
< jnewbery>
aj: I think that's effectively what 20284 does
< sipa>
can we imagine any backward but not forward compatible change?
< vasild>
aj:
< vasild>
16:15 < vasild> pr20284 makes it possible to say in peers.dat "this is format=5, but anybody who can
< vasild>
read format=3 can also read this one"
< aj>
vasild, jnewbery: i believe you're both mistaken... it just supports a range: "this code supports versions 3..5". introducing version 6 won't make old code support it
< jnewbery>
sipa: after #19954, all changes are not forward compatible (because 0.21 onwards will reject any version greater than the one they know)
< aj>
sipa: backward compatible is easy: if version == 4: ... elif version == 5: ...
< jnewbery>
aj: +1
< sipa>
jnewbery: yes, i mean semantically
< sipa>
is there any change we can imagine to peers.dat that would remain readable by new versions?
< aj>
sipa: readable by old code, you mean?
< sipa>
oh
< * sdaftuar>
is super confused
< sipa>
do i have my forward/backward mixed up?
< aj>
new code, old peers.dat == backwards compatible
< sdaftuar>
sipa: i thought john has his forward/backward mixed up, for what it's worth. and maybe aj too.
< aj>
old code, new peers.dat == forwards compatible
< vasild>
awkward
< sdaftuar>
vasild: +1
< aj>
"the format is ___ compatible" is how i'm interpreting it
< sipa>
the goal here is to make sure that new versions can keep reading old files?
< jnewbery>
sdaftuar: wikipedia seems to agree with our definitions: Backward compatibility (sometimes known as backwards compatibility) is a property of a system, product, or technology that allows for interoperability with an older legacy system, or with input designed for such a system
< vasild>
no
< jnewbery>
where the input here is peers.dat
< vasild>
sipa: new code can always read old files
< jnewbery>
old peers.dat, new addrman is backwards compatibility
< jnewbery>
new peers.dat, old addrman is forwards compatibility
< sipa>
ok so this is about new files being readable by old code? can we imagine a change where that's useful?
< sdaftuar>
sipa: it's about being able to downgrade
< sdaftuar>
and not losing all your data
< jnewbery>
i.e. you right the code in a way that it can handle future changes to format that you don't know about, which is potentially important for downgrades
< sipa>
sdaftuar: yes, my question is: can we imagine any change to peers.dat where that's even feasible?
< luke-jr>
sipa: if you don't use Tor, this change is?
< sdaftuar>
sipa: well it depends on what's on the table? eg if we wrote out the old format for the addresses that are supported by the old format in a file that is readable by old software, then sure
< jnewbery>
yes, for example the change from 0 to 1 and 1 to 2 both allowed old versions to read new files
< sipa>
jnewbery: ah, indeed!
< sipa>
thanks
< vasild>
sipa: you authored that! :-)
< luke-jr>
hmm, we could in theory even split a peers-torv3.dat for those, and keep peers.dat as-is and compatibel?
< sipa>
vasild: it's early
< sdaftuar>
luke-jr: interesting idea
< aj>
or put the torv3 addresses as an add-on at the end of the file or something?
< wumpus>
where does that stop, though... would we have peers-i2p.dat as well?
< jonatack>
if anyone following along is wondering, we're talking about enum class Format in addrman.h::L269
< luke-jr>
wumpus: why not?
< sipa>
this seems hard
< luke-jr>
is there a reason to throw it all in the same file?
< jnewbery>
jonatack: well more generally about the Serialize and Unserialize methods for CAddrMan
< sdaftuar>
wumpus: perhaps eventually files get merged, once old software no longer is a plausible downgrade target?
< sipa>
peers.dat files aren't just lists of addresses
< sipa>
they dump the tables and their layout too
< jnewbery>
we could replace peers.dat with sqlite :)
< wumpus>
I'm really not sure this is worth so much worrying about, if we're really worried about people downgrading why not make a backup at first run with 0.21.0?
< wumpus>
then if people downgrade they can restore the backup and *shrug*
< sipa>
if you split the file in two, you can't just merge them back without losing informatiom
< luke-jr>
jnewbery: that might not be a bad idea
< sdaftuar>
wumpus: yes i agree with that type of suggestion too, along the lines of "sometimes it can be annoying to downgrade"
< luke-jr>
wumpus: how is that different from just using a new filename, except requiring an extra step to downgrade?
< wumpus>
luke-jr: it keeps the filename the same for the bulk of the users
< wumpus>
the non-downgrading path is likely be far most popular, or at least one'd hope :)
< wumpus>
in any case, we needa solution here fairly quickly
< jnewbery>
wumpus: I agree that it's more likely, but I think we should strive to make downgrades across a single version seamless, just in case we ship a bad bug and need people to roll back
< jnewbery>
*upgrading is the more likely path than downgrading
< jnewbery>
ok, we're running out of time and there are a couple more topics, so perhaps we should move on
< wumpus>
I'm not sure last-minute changes lke 'sort peers.dat per network' or other complete changes to handling peers database
< sipa>
wumpus: agree
< jnewbery>
vasild: did you have anything you wanted to add about I2P support?
< ariard>
jnewbery: I'm fine to report mine next time
< jnewbery>
ariard: thanks. Maybe a good idea so people have some time to think about it before the meeting
< sdaftuar>
i've got a question about i2p (applies to any new exotic network, even tor stuff that we're doing) - what is our undestanding for how these addresses get gossiped to peers that support the network type?
< wumpus>
jnewbery: yes, agree, it's kind of nasty that this issue comes up so last minute
< sipa>
sdaftuar: i2p/cjdns do not get rumoured by the current code, at all
< vasild>
sipa: no, they would get rumored if the node has connectivity to i2p
< vasild>
or am I mistaken...
< wumpus>
cjdns not because it uses IPv6 addresses in a range we consider local / unroutable?
< sipa>
vasild: which isn't possible.on the current code :)
< vasild>
btw, in I2P we can see the peer's address and be sure data comes from him (the address is a hash of his public key)
< sdaftuar>
sipa: thanks. one more question, how do we decide what address to advertise ourselves as, if we're reachable in multiple ways?
< luke-jr>
vasild: same for Tor? :p
< vasild>
luke-jr: no, in Tor we don't see the address of the peer that connected to us
< luke-jr>
oh, right
< sipa>
"tor has no from address"
< luke-jr>
XD
< wumpus>
heh
< vasild>
so in I2P we have P2P encryption by default :-)
< jnewbery>
sdaftuar: that logic is in AdvertiseLocal() in net.cpp
< sipa>
vasild: with public identities, though :(
< vasild>
yes
< luke-jr>
kinda by design in i2p?
< vasild>
yes
< sipa>
sdaftuar: among the local addresses compatible with that peer, the one that has received the most mentions in incomimg version messages
< sipa>
which reminds me: we can't put torv3/i2p/cjdns in version messages, so those will never receive any mentions
< sdaftuar>
sipa: it seems like for something like adding a new newtork type, sometimes we'll want to advertise our i2p address instead right?
< sdaftuar>
sipa: oh taht seems important
< jnewbery>
do we need a new version version? :)
< sdaftuar>
jnewbery: i think we can just add data on to the end and no one will mind
< sipa>
i'm not sure this is useful for those networks
< sipa>
as there isn't any useful compatibility matrix for them
< jnewbery>
that's time!
< jnewbery>
#endmeeting
< sdaftuar>
sipa: address rumoring isn't useful, or something else?
< jonasschnelli>
oh.. I missed the meeting.
< wumpus>
sipa: there was the idea to add that address in the 'sendaddrv2' message
< vasild>
I guess sipa means the "voting"
< sdaftuar>
ah
< vasild>
or how many times the address receives "mentions"
< aj>
zzz
< * luke-jr>
gives jonasschnelli the mic
< jonasschnelli>
I wanted to ask about BIP324's rekeying
< sipa>
sdaftuar: scoring your local addresses using incoming version mentions isn't useful for these separate networls
< sipa>
jonasschnelli: ongoing discussion, it seems!
< sdaftuar>
yeah i was thinking that if we are trying to bootstratp a new type of address, we need to proactively advertise that address even to peers who don't understand it
< jonasschnelli>
sipa: may you have 10 minutes for discussion the 1s rekey approach
< sdaftuar>
so that eventually peers that do support it, receive it
< sipa>
sdaftuar: that's a good point
< wumpus>
sipa: (which would be easier to add new things to than the version message, and is relevant to the same functionality)
< vasild>
sdaftuar: my thought exactly, but we decided to not relay i2p addresses by nodes that are not connected to i2p; at least in 0.21
< jonasschnelli>
sipa: indeed... its ongoing. But assume a 1s rekeying interval would imply some sort of a rekey-message,.. right?
< sipa>
jonasschnelli: my current thinking is that doing rekeying at the key stream level is super cheap and simple
< jonasschnelli>
rekey on every ping/pong would be less complicated to implement (rather then time based)
< sipa>
jonasschnelli: and i see no reason to have it at the application level as well if we do that
< sipa>
jonasschnelli: i absolutely don't suggest rekeying every secomd :)
< jonasschnelli>
sipa: at stream level would be something like basically rekey on after message?
< vasild>
there is already one bitcoin node listening at yfsvsy467mt5xafaq7zaukkjyzehvmew445yaaejvrwpk53acejq.b32.i2p (irc gossip :-D)
< jonasschnelli>
"after every message
< sipa>
jonasschnelli: no, every X bytes
< sdaftuar>
vasild: nice :)
< jonasschnelli>
sipa: okay. What we have now.. but probably smaller than 1GB
< sipa>
jonasschnelli: no!
< sipa>
it's currently done at the layer above
< sipa>
this would let you just build it inside chacha
< jonasschnelli>
ah. Built into the AEAD
< ariard>
jonasschnelli: I think you can pick up a value based on blocks frequency if you want something to happen every Y period of time?
< sipa>
without having a bit to signal rekeying or so
< sipa>
ariard: that's just as hard
< sipa>
jonasschnelli: this would give you forward secrecy (also a weird term
< jonasschnelli>
What if the byte limit hits in the middle of a message payload?
< sipa>
for ~free, at the byte level if you wanted to
< sipa>
jonasschnelli: yes, so what?
< sipa>
the stream cipher does it automatically
< ariard>
sipa: do you mean byte-accounting is hard to get right?
< sipa>
ariard: no, byte-accounting is trivial
< jonasschnelli>
sipa: hmm... but the mAC?
< sipa>
jonasschnelli: the mac keys don't cycle
< sipa>
only the encryption keys
< jonasschnelli>
but the current MAC key is derived from the chacha20 stream
< sipa>
jonasschnelli: yes that'll need to change
< jonasschnelli>
sipa: hmm...
< sipa>
have a separate stream where the mac keys are drawn from, for example
< sipa>
but it means you can drop the signalling of rekeying etc
< sipa>
and concerns about peers not doing it, or doing it too often
< sipa>
as long as we don't worry about secrecy in the other direction (which the current bip324 writeup doesn't, afaik), i think this is very much preferable
< jonasschnelli>
sipa: do the benefits of this justify further deviation from the "well tested" ChaCha20Poly1305@openssh?
< sipa>
jonasschnelli: we're already deviating from it
< jonasschnelli>
sipa: but in a pretty minor way
< sipa>
jonasschnelli: maybe superficially... but openssh rekeys by redoing DH, no?
< jonasschnelli>
sipa: what do you think are the benefits over a rekeying signal (where we can detect DoS or exceeded limits)?
< jonasschnelli>
sipa: probably... haven't looked at their rekeying.
< jonasschnelli>
but the AEAD has no rekeying (@openssh and @Bitcoin)
< sipa>
so doing rekeying by redoing DH makes sense, as it gives you secrecy in both directions
< jonasschnelli>
Yes. Our rekeying does a independent rekey per direction,... whenever the limits are reached.
< sipa>
rekeying by hashing only gives you forward secrecy, not the other direction
< sipa>
but as long as we only care about forward secrecy... doing the whole signalling thing is kind of overkill
< jonasschnelli>
I see
< jonasschnelli>
I see what you mean with direction noew
< jonasschnelli>
*now
< sipa>
you can get cheap forward secrecy at the byte level, instead of at the 1 GB level
< sipa>
(by generating 4096 bytes of stream cipher ahead of time, rekeying, and then throwing the stream cipher bytes away as they get used)
< sipa>
you can't do that with 1 GB of material, and can't do that if you don't know when the other party is going to reky
< sipa>
anyway, my point is: not redoing DH is already a big conceptual departure from the existing protocol
< jonasschnelli>
I think I understand this... I'm just unsure about the MAC construct... and concerned about complication of the implementation (with its risks)
< sipa>
yeah
< sipa>
to be honest, i habe difficulty remember the exact way the stream/keys are constructed now too
< sipa>
this may simplify it...
< jonasschnelli>
sipa: we rekey with SHA256(SHA256(session ID || old_symmetric_cipher_key))... wouldn't an attacker stealing the symmetric be unable to go in both directions?
< jonasschnelli>
since the "session ID" requires the ECDH key
< sipa>
jonasschnelli: if an attacker knows the old key, they can also apply the hashing
< jonasschnelli>
sipa: the attacked doesn't know the session ID, right?
< sipa>
well in this scenario they do!
< jonasschnelli>
sipa: So the scenario is that ECDH is broken?
< sipa>
jonasschnelli: or they broke into your machine and dumped its memory
< sipa>
forward secrecy protects against that case... if an attacker does that, they *still* can't decrypt old messages
< jonasschnelli>
okay.. got it
< sipa>
but the other direction... you can't prevent an attacker who read your RAM from decrypting future messages, unless you redo DH
< jonasschnelli>
including re-generation of emphemeral keys,.. right?
< sipa>
yes, that's what DH does
< jonasschnelli>
indeed. :)
< jonasschnelli>
and you think that backward secrecy (probably called different) can be achived by altering the AEAD and including a deterministic rekeying?
< jonasschnelli>
or has the AEAD-inner-rekeyng nothing to do with the secrecy directions?
< sipa>
that only provides forward secrecy
< sipa>
but it can do it at the byte level
< jonasschnelli>
more elegant but same level of security as the current proposal?
< sipa>
indeed
< jonasschnelli>
okay... got it.
< sipa>
perhaps we want some affordance for redoing DH too, but we can probably just disconnect and reconnect for that...
< jonasschnelli>
I haven't seen BIP324 discussion about the missing backward secrecy of the current rekeying approach.
< sipa>
i think it's less of a concern
< sipa>
but i'll try to summerize
< sipa>
on githun
< jonasschnelli>
great,... thanks.
< jonasschnelli>
I'd like to not loose to much momentum on the implementation. But also,... not urging to stop improving the protocol. :)
< jonasschnelli>
However... thanks sipa for explaining!
< wumpus>
vasild: here is already one bitcoin node listening at yfsvsy467mt5xafaq7zaukkjyze hvmew445yaaejvrwpk53acejq.b32.i2p (irc gossip :-D) <- good to know, i'll have a look at setting up I2P too
< jonatack>
vasild: i think i'm connected to your i2p peer
< amiti>
hi! does anyone actively use `-loadblock` / have current use cases?
< sipa>
amiti: did yiu see my response on the issue?
< amiti>
yeah, you said "I don't know if those see much use still", so I thought I'd ask more people :)
< sipa>
i mean: it was intended for supporting bootstrap.dat, and i think that's the only use case
< sipa>
if there is anyone actively relying on that feature i don't know
< amiti>
gotcha. ok maybe I could rephrase the question as- does anyone actively use bootstrap.dat?
< luke-jr>
it's not even maintained anymore
< sipa>
not by us
< sipa>
though some sites provide them
< luke-jr>
updated? O.o
< sipa>
since the improved block download logic in 0.10 there is much less use for it, as just syncing from random peers is likely faster than downloading + processing in sequence
< luke-jr>
I suppose it might be handy for people syncing offline PCs
< sipa>
TIL also
< luke-jr>
hmm, I should probably stop parsing my entire debug.log every hour
< luke-jr>
it's getting behind XD
< amiti>
do I understand correctly: the idea of bootstrap.dat / loadblock is that you input a file with the sequential blocks to process. in the current state, what are the differences from importing the datadir and reindexing? / would there be any advantage? (ofc one difference is that loadblock requires the blocks to be in order)
< sipa>
i expect you can just rename a bootstrap.dat file to be blk00000.dat, and run -reindex
< sipa>
amiti: i think part is that you don't want to encourage people copying entire datadirs, as chainstates don't get re-verified
< sipa>
and anything that involves manually messing with the datadir is probably reserved for people who know what they're doing anyway
< amiti>
oh I see, so loadblock does more verification than reindexing? ok I'm not super familiar with reindexing so I'll go learn more
< luke-jr>
loadblock just treats the file as an untrusted peer
< sipa>
amiti: no, it's exactly the same
< sipa>
amiti: but if you copy a *chainstate* from someone (not just block files), they don't get re-verified (because your node wouldn't know it's operating on an imported copy)
< amiti>
ahhhh I see
< sipa>
still, i don't have a good answer for whether it's worth supporting -loadblock if it is a burden to maintain... i suspect not
< sipa>
i also don't think it's a large burden, thoug
< * midnight>
makes use of loadblocks still fwiw
< sipa>
good to know!
< amiti>
yeah, for me the first question is "is it still useful". then the next question is "is it a higher burden to maintain, or to remove?"
< sipa>
i don't think i've personally used it for many years
< amiti>
midnight: ooo, willing to tell us more about how/why you use it?
< midnight>
sure. I use it as a way of explicitly migrating block data between machines; also occasionally to (expensively) explore early blocks and early large-block-number reorgs. Mostly as a way to ensure migrated block data is reverified with modern versions as my node instances are typically very long-lived.
< midnight>
I'd be completely fine with just a network-based block injector if it exists.
< amiti>
okay gotcha. thanks!
< midnight>
\o I wouldn't be too broken-up about it being removed, as long as it's still possible for me to help seed my friends with a copy of current block data from my own nodes somehow.
< midnight>
(so, I test it first before handing it to them)