< bitcoin-git>
bitcoin/master a6a1b10 fanquake: guix: only download sources for hosts being built
< bitcoin-git>
bitcoin/master 3371758 fanquake: Merge #21078: guix: only download sources for hosts being built
< bitcoin-git>
[bitcoin] fanquake merged pull request #21078: guix: only download sources for hosts being built (master...guix_selective_download) https://github.com/bitcoin/bitcoin/pull/21078
< gleb>
jnewbery: Could you give me a slot at the start to talk about erlay update? No discussion, just updating folks. I can only attend only the first half of the meeting that's why asking in advance
< jnewbery>
gleb: of course
< gleb>
thank you!
< wumpus>
fanquake: AKZ98 is a new spammer? (what a coincidence just after banning the last one)
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #21279: scripted-diff: Regenerate key_io data deterministically (master...2102-testDetGen) https://github.com/bitcoin/bitcoin/pull/21279
< wumpus>
it follows the same naming scheme as PS113, so i guess it's the same spammer and it's coordinated somehow, we might want to contact github admin adbout this
< linkrage>
Hi, I've just compiled bitcoin-0.8.6 from source on Ubuntu 14.04 (on purpose). It doesn't build bitcoin-cli and looks like that's on purpose ? How dow I communicate with bitcoind to set a passphrase on wallet etc. - curl only ?
< gleb>
I’m almost done with that, just need to split cpp/header, split into commits, and add a bit more comments. Gonna be ready in the next couple days.
< gleb>
jnewbery jonatack amiti I recall you mentioned you’d prefer a more modular appoach. Would any of you be able to take a look during this week, before I update the main PR?
< aj>
jnewbery: good nachos to you too!
< gleb>
I would really appreciate initial feedback on the new modularity, and I think this would also help to save review time of those folks focusing on other aspects of the PR.
< gleb>
It’s just that if this new approach is also not really good (for now I personally think it is good), I’d prefer us realizing it earlier, and re-shaping it again.
< prayank>
jnewbery: TIL aj is working on Dandelion
< jnewbery>
gleb: sure, I'm happy to take a look at it
< gleb>
jnewbery. Great, thanks. If anyone else is willing to join john, please let me know :)
< gleb>
Otherwise, that's it.
< jnewbery>
where is it? how do you want me to leave review?
< amiti>
gleb: I'm also down
< gleb>
It's under naumenkogs/bitcoin/tree/erlay_refactored, but it's currently not ready yet, as I said.
< jnewbery>
I also still think PRing minisketch separately would help move things forward
< sipa>
perhaps leave a comment on the PR when you're done with the refactoring and want comments?
< ariard>
or leave a message on irc here
< gleb>
jnewbery: yeah, since c++ headers is merged, we can probably PR minisketch
< gleb>
sipa: I'm wondering if you'd be a better candidate for merging minisketch? Just realistically, discussing the code and such
< gleb>
Or you'd rather me do it?
< gleb>
Not merging but creating PR I mean.
< sipa>
i'll open a PR to fix the issue that the current minisketch tests run forever (which is unexpected and confuses people)
< sipa>
i think after that we could
< sipa>
i
< sipa>
i'm happy to PR it
< gleb>
Great, thank you.
< gleb>
I'll focus on the erlay part then
< jnewbery>
would minisketch tests run with `make check` in the bitcoin core?
< sipa>
they probably should
< sipa>
(and that makes it a blocker to not have them run forever...)
< gleb>
jnewbery amiti: Let's figure the best way you can help with my erlay refactor in personal messages once I'm ready
< jnewbery>
gleb: just leave a comment on the PR when it's ready and I'll take a look at the branch
< amiti>
+1
< sipa>
+1
< gleb>
alright
< jnewbery>
that PR is a bit unwieldly at this point. If you're changing the approach and minisketch is going to be split out of it, it might make sense to start a new PR, but you can decide that later
< gleb>
jnewbery: I totally agree.
< aj>
(+1 on a new PR to avoid all the load mores)
< sipa>
yeah
< jnewbery>
ok, we have three other topics:
< jnewbery>
- Orphan reprocessing (jnewbery)
< jnewbery>
- Peer rate-limiting (jnewbery)
< jnewbery>
- Dandelion update (ajtowns)
< jnewbery>
anyone want to add any quick updates or other topics before we move onto those?
< jnewbery>
Dandelion update sounds like it might be the quickest, so maybe we do that first
< aj>
oh, i was happy for it to be the if-we-had-time
< aj>
anyhoo
< aj>
we were talking about dandelion last week, and maybe have some ideas about how to do it without needing a separate stempool. but it was pointed out there was an idea to do a -lite variant, that only stems for a single hop
< aj>
that has weaker privacy support, but doesn't need new p2p messages or a separate mempool, so should be much easier to implement
< glozow>
does one-hop work? couldn't you tell where a tx originates if you knew they were one-hopping to you?
< glozow>
idk if i'm missing something
< aj>
i've been chatting on twitter with gulia and she's updated her sims
< gleb>
do you make a new connection for one-hop, or pick from existing conns?
< jnewbery>
I think if an attacker made multiple connections to you, they could tell if you're only relaying a transaction to a single peer
< aj>
glozow: one-hop stem == pick one or two outbound peers and send your tx to them; when you receive a tx always flood it to everyone. so you don't immediately know if a tx was stemmed/flooded to you (and hence don't need different messages)
< aj>
jnewbery: if an outbound connects back to you as an inbound, yeah
< aj>
jnewbery: (you don't stem to inbounds to make that harder, i think)
< aj>
it looks to me like -lite is a signficant enough improvement that it's probably worth tryng, and dandelion++ proper is enough of an improvement on that that it's still worth thinking about if we can make it feasible
< prayank>
aj: I am also trying to resolve conflicts and add Dandelion code (from BIP implementation link) in current master branch. I wasn't even aware you are working on it because nobody informed on IRC, Github or SE when we discussed Dandelion. I will read the Twitter thread and DM you later.
< sipa>
prayank: dandelion lite has basically nothing to do with dandelion (in terms of implementation complexity)
< aj>
prayank: there's a link in that twitter thread to the irc conversation last week
< gleb>
aj: Erlay does something really similar to lite, actually.
< glozow>
i don't think the bip is updated for dandelion++
< glozow>
is it?
< prayank>
sipa: Okay
< sipa>
the bip is dandelion++ afaik
< sipa>
my impression from dandelion lite is that it's basically 1% of the work to get 80% of the benefits (under vaguely reasonable assumptions)
< aj>
sipa: looks more like 40%-50% of the benefits to me
< sipa>
okay.
< ariard>
so the model is alice initial-broadcast to bob, it's testmempoolaccepted to not let any fingerprint in bob's mempool and stems to caroll, an outbound?
< glozow>
i think bob floods, no?
< lightlike>
would it be either dandelion or erlay, or would both work together?
< aj>
for lite, bob floods
< sipa>
there is no stempool in lite, right? you just remember in the wallet itself, and the rebroadcast uses the full broadcast
< ariard>
okay gotcha
< gleb>
lightlike: they should work together just fine
< glozow>
what if bob black-holes it?
< glozow>
doesn't relay it, i mean
< sipa>
glozow: rebroadcast takes care of it, just like in dandelion (correct me if i'm wrong, aj, i'm going from very vague memory)
< aj>
glozow: you hit the wallet rebroadcast timer and do a flood, probably
< glozow>
doesn't rebroadcast not run for 12 hours at least?
< jnewbery>
can you summarise what the 40-50% benefit is?
< aj>
sipa: i don't think those details are really specced anywhere
< gleb>
black holes also can be detected and disconnected after one occasion.
< aj>
jnewbery: the statistics used are "precision" and "recall" which measure how likely an attacker is to be able to pinpoint the originator of a tx
< jnewbery>
why is relaying to one outbound peer better than relaying to 2, or 8?
< sipa>
glozow: that'd need to be adjusted then; iirc dandelion has a delay of just a few seconds
< gleb>
jnewbery: other 7 of your conns + inbounds have worse idea that it originated from you
< gleb>
Rather than when you relay to all of them
< amiti>
I think an implementation of dandelion lite would want to add logic for checking if the txn successfully relayed / a quicker rebroadcast if not.
< aj>
jnewbery: stemming to one outbound peer is what's modelled in the sim and paper; i don't think it's clear if there's a best value to use
< glozow>
sipa: sorry i meant, i think currently rebroadcast is on 12-hour-ish timer, but dandelion++ spec has a several second timer. trying to wrap my head around what needs to be implemented for dandelion lite, perhaps would need to have a timer for rebroadcast-just-incase
< ariard>
amiti: quite easy by probing your own mempool to see if you get an announcement back?
< amiti>
yup, exactly
< gleb>
jnewbery: oh sorry, I guess you ask about the exact choice, yeah, 2 could be better than 1..
< amiti>
that just doesn't exist yet
< gleb>
glozow: I was also wondering if rebroadcast here should happen in 10 seconds
< sipa>
glozow: this would be a separate rebroadcast mechanism i believe
< gleb>
I don't see why we should wait 12 hours?
< amiti>
aj: is it easy to run the simulation with 2 outbounds and compare?
< sipa>
because the dandelion lite "stem" relay can't insert into the mempool
< glozow>
gleb: I mean the current rebroadcast behavior on master, without adding any dandelion stuff
< ariard>
i think the rebroadcast should be considered in function of opening a one-shot for the connection
< aj>
amiti: the sim chooses an anonymity graph with 4 outbounds for each node, and selects one of those outbounds for relay. changing 4 to 2 or 8 is easy, but selecting more than one outbound requires code changes...
< gleb>
ariard: one shot wouldn't work at all, unless nodes don't do that for foreign transactions?
< gleb>
because it's obvious one-shot is a tx source
< gleb>
unless tor
< aj>
gleb: yeah, tor or i2p for oneshot
< amiti>
aj: gotcha
< gleb>
aj: sorry, is this relevant? Are we talking about oneshot over tor as a way to do d lite?
< ariard>
gleb: don't we already assume bob has an easier job to guess alice is the source if you mass-connect?
< gleb>
ariard: Sure, but lite would help against that even without one-shot tor
< aj>
gleb: i thnk the idea is dandelion-lite should work even if your node doesn't do tor/i2p at least. (people talk about lots of things :)
< gleb>
aj: without oneshot then. oneshot without tor doesn't work.
< aj>
gleb: yep
< jnewbery>
aj: do you need help with anything? Would you like people to review a proposal or something else?
< aj>
jnewbery: no, just wanted to update people on what's going on. i don't have a particular plan on what's next at this point
< ariard>
gleb: could you do one shot without tor for a low % of foreign transactions ?
< amiti>
aj: so do I have this right that your current impression that we could implement lite without any protocol changes, code not being too invasive, and get ~40-50% of benefits? what are next steps?
< amiti>
oh, you just answered part of that
< gleb>
I'd be willing to help. And as a separate thing, aj, we should see how to marry it with erlay in practice, once you reach that point.
< gleb>
ariard: that's something was never discussed. I see that as a measure *on top*, but I'd not rely on this solo.
< aj>
amiti: yes, i think that's right. not sure if more sims are worth doing first; otherwise figuring out how to be able to stem to some peers without contaminating how you respond to the peers you don't stem too would probably be the first bit of implementation
< jnewbery>
aj: hopefully shouldn't be too difficult to update FindTxForGetData. We've done something similar before
< aj>
jnewbery: i think working out the spec would be harder than implementing it...
< sipa>
haha
< gleb>
ariard: that's a cool idea to add noise against any kind of spying. But i think the problem with oneshot-no-tor is that it's success is a function of how ready we are to dos our own network :)
< jnewbery>
yup
< amiti>
jnewbery: something similar? what are you thinking of?
< jnewbery>
m_recently_announced_invs
< jnewbery>
maybe it just works already
< ariard>
gleb: yeah I know...kinda some trade-offs we're hitting with new rebroadcast, more noise, more bandwidth
< jnewbery>
we currently reprocess orphan transactions in the context of the node that provided the parent
< jnewbery>
ie node A sends us an orphan transaction, node B sends us its parent, we later process the orphan transaction next time we call ProcessMessages() for node B
< jnewbery>
it seems a bit strange that we do this, and aj was proposing we change that as part of, or after, orphan handling is split into txorphange
< ariard>
idea would be to assign orphan processing to A only?
< sipa>
so that means we may punish the peer that provided parent for an invalid orphan received from someone else?
< jnewbery>
that seems like a reasonable change to me. We should either process orphan reprocessing in the context of node A, or in some global context (i.e. call PeerManager.DoGenericWork())
< aj>
today, we still lookup the original peer when figuring out who to punish
< jnewbery>
sipa: no, we already punish the peer who provided the orphan if it's consensus invalid
< sipa>
then what do you mean by "in the context of" ?
< ariard>
hmmm I would rather avoid global context where you can trace back and punish dosy peers
< jnewbery>
sipa: we process the transaction the next time we call ProcessMessages(node B)
< ariard>
*can't
< aj>
sipa: we process the tx when we call ProcessMessages for peer B at present (because the child tx goes into the orphan work set of whoever provided the parent)
< amiti>
I think "in the context of" refers to which node we allocate ProcessMessage time to
< sipa>
ah, i see
< aj>
the idea would be to move the orphan work sets out of the Peer mapping and into a new map in the txorphanage, which can just be sorted by the frompeer, and an entry selected from it during frompeer's ProcessMessages
< aj>
moving the orphan work sets out of Peer also allows g_cs_orphans to be moved into txorphanage and stop being a global
< jnewbery>
aj: I like this change a lot :)
< jnewbery>
my last proposal was to move orphan reprocessing to a global work queue: ttps://github.com/bitcoin/bitcoin/pull/19364 , but moving it to the work queue of the peer who sent it also seems fine
< amiti>
is the core reasoning to prevent the sorta attack mentioned in this comment: https://github.com/bitcoin/bitcoin/pull/21224#issuecomment-781741925, where a peer who sends a valid txn that activates lots of invalid orphans would take up our CPU time instead of the one who will eventually be punished?
< ariard>
amiti: I'm not sure the peer who sends a valid txn can be actually the culprit for it
< amiti>
ariard: what?
< ariard>
a malicious peer, if knowledgeable of tx-relay link between two peers might provoke mempool conflicts to trigger this scenario...
< ariard>
le'ts say you have alice connected to bob, and mallory to both of them, you send tx A to alice, tx A' to bob
< sipa>
jnewbery: i haven't paid attention to the latest discussion; i just hope the motivation is more than "this results in cleaner code for us" if changes observable behavior
< ariard>
you send a set of childrens from tx A, will be relayed to bob and stored as orphan
< ariard>
receiving orphans on bob side, should trigger him to fetch again tx A and thus do the work on alice account
< aj>
(i think it's "add latency to Alice's transactions" more than "use up Alice's CPU time")
< amiti>
aj: ok, yeah, that makes sense
< amiti>
ariard: I think you're saying that an honest peer can send a parent to invalid orphans received from a different peer?
< ariard>
aj: do we actually have this notion of peer CPU time? I don't think so...
< jnewbery>
ariard: not yet. That maybe leads nicely into the next topic
< ariard>
amiti: orphans can be valid there too, and it can work as received from a different peer but not exactly the scenario I was describing
< jnewbery>
Again, this came up in the discussion of #21224
< aj>
ariard: Mallort constructs tx A, and invalid children of A: B1, B2, B3, B4, ... Mallory relays B1..Bn to Bob. Mallory relays A to Alice. Once Bob receives A from Alice, he'll have high latency for the next tx from Alice
< ariard>
amiti: the scenario described by aj is the one you were thinking of, mine orphans are announced to alice (note they're considered as orphan but valid txn by alice)
< ariard>
AFAICT
< jnewbery>
aj: I agree. It makes perfect sense that we don't slow down processing for Alice because of transactions that Bob relayed
< ariard>
*are not considred as orphan (damn english)
< jnewbery>
*that Mallory relayed
< aj>
ariard: bob and alice will relay A and A' to each other, reject them as double spends, and then Alice will reject the children rather than treat them as orphans because they have a rejected parent, i think?
< ariard>
aj: at first sight but you can play with rbf and add a grandparent to A and A' to make it work
< amiti>
yeah, the scenario aj described is my understanding of the reasoning behind why we would change which peer context we process a parent in.
< ariard>
aj: or if A' has been announced through wtxid-relay are we going to add to rejection filter its txid? orphan parent are fetched through txid
< jnewbery>
ok, that's time!
< ariard>
aj: I should craft a functional test with the scenario, easier way to debate about it :)
< aj>
ariard: we don't reject a tx until we've received it (the tx, not its w/txid), and we add both txid and wtxid to recentRejects?
< aj>
or rather we add the txid to recentRejects as well when we know it will fail no matter the witness. maybe that's not the case for double-spends, because theoretically there could be a smaller valid witness that would meet RBF rules?
< ariard>
aj: nope smaller witness valid are bounce back by current mempool logic, see #19645
< sipa>
jnewbery: fwiw, i think that arguments about the ability for bad latency for processing of messages from unexpecting peers is a totally valid reason to change behavior
< sipa>
sorry for the kneejerk reaction when this was brought up again
< aj>
ariard: no, i mean if you've got tx A, and want to RBF it to tx B, but B's fee isn't high enough, then tx B' with the same txid but smaller witness data might still be accepted. If you did accept B, you won't accept B' as you point out
< aj>
sipa: sounds like it needs a PR so we can do less hand-wavy review though
< ariard>
aj: okay thinking further, my naive scenario don't seem to hold but yours where mallory directly announce B1,...,Bn to Bob and relay A to Alice should work
< ariard>
we do AlreadyHave() at orphan parent fetching