hernanmarino_ has quit [Ping timeout: 248 seconds]
hernanmarino has joined #bitcoin-core-dev
l0rinc has quit [Quit: l0rinc]
jadi has quit [Ping timeout: 256 seconds]
Cory56 has quit [Quit: Client closed]
Cory56 has joined #bitcoin-core-dev
l0rinc has joined #bitcoin-core-dev
l0rinc has quit [Client Quit]
l0rinc has joined #bitcoin-core-dev
l0rinc has quit [Quit: l0rinc]
entropyx has quit [Quit: no love here!]
jadi has joined #bitcoin-core-dev
jadi has quit [Ping timeout: 256 seconds]
l0rinc has joined #bitcoin-core-dev
<_aj_>
sipa: "time_since_last_send * R" not "average_time_between_sends * R" ?
<_aj_>
sipa: if we're happy to increase our INVs quadratically when bursts come in, why not just make the formula be min(1000, inv_to_send.size()) and always send as much as we can? (i think the answer is "to smooth out bursts of low-fee txs so as to avoid wasting the network's validation and relay resources"; and i'm not sure a quadratic increase during bursts fits well with that?)
kimsh has joined #bitcoin-core-dev
<sipa>
_aj_: yeah, time since send, not avg time between sends
<sipa>
and the quadratic term is essentially the "give up on trying to smooth out if it causes too much backlkg"
<_aj_>
"the reason the current formula is so bad" -- i don't think it's "so bad"; i'm only seeing occassional queues of 5000 txs from peers that don't seem to be announcing txs back to me. the original issue was we apparently had a 6h period of sustained 13.9 tx/s announcements of BRC20 things, and i think i was seeing 25k to 60k+ queued txs by some peers; current behaviour is much milder.
<sipa>
yeah, i don't think it's bad in practice - but in theory it can grow very large still i think (but far less than before the /1000*5 term got added)
<sipa>
_aj_: the effect of the quadratic is really "if you're within x% of max queue size, send x% of the actual queue"
<_aj_>
yeah; with normal traffic (<7tx/s) any backlog gets cleared in ~15 minutes
<corebot`>
https://github.com/bitcoin/bitcoin/issues/7840 | Several performance and privacy improvements to inv/mempool handling by sipa · Pull Request #7840 · bitcoin/bitcoin · GitHub
twistedline has quit [Ping timeout: 258 seconds]
PaperSword has quit [Ping timeout: 244 seconds]
roconnor has quit [Ping timeout: 260 seconds]
l0rinc has joined #bitcoin-core-dev
l0rinc_ has joined #bitcoin-core-dev
l0rinc has quit [Ping timeout: 256 seconds]
roconnor has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
hugohn______ has quit [Server closed connection]
hugohn______ has joined #bitcoin-core-dev
Cory56 has quit [Quit: Client closed]
Cory56 has joined #bitcoin-core-dev
<_aj_>
sipa: hmm, what if we tracked the exponential rolling average of relayed txs; `past_avg /= pow(2.0, timedelta/5m); ++past_avg;` in RelayTransaction() or so. could then make our broadcast target be min(queue, max(TARGET_RATE, past_avg), 1000)
l0rinc_ has quit [Quit: l0rinc_]
l0rinc has joined #bitcoin-core-dev
mbrochh has quit [Quit: Connection closed for inactivity]
QUBIT_ABHAY has joined #bitcoin-core-dev
QUBIT_ABHAY has quit [Client Quit]
QUBIT_ABHAY has joined #bitcoin-core-dev
QUBIT_ABHAY has quit [Ping timeout: 250 seconds]
f321x has joined #bitcoin-core-dev
Guyver2 has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
robszarka has joined #bitcoin-core-dev
szarka has quit [Ping timeout: 256 seconds]
l0rinc has quit [Ping timeout: 256 seconds]
hirish_ has quit [Ping timeout: 260 seconds]
iSiUp has quit [Quit: WeeChat 4.7.1]
hirish has joined #bitcoin-core-dev
iSiUp has joined #bitcoin-core-dev
l0rinc has joined #bitcoin-core-dev
jerryf_ has joined #bitcoin-core-dev
jerryf has quit [Ping timeout: 272 seconds]
zeropoint has quit [Quit: leaving]
f321x has quit [Quit: f321x]
<fanquake>
Does anyone want to look at #32097? We are going to ship Linux GUI binaries that aren't working properly (on modern desktops), and already are shipping GUI binaries that may/may not work
<fanquake>
Ideally we'd be shipping GUI bins that didn't require users to manually install missing dependencies
<fanquake>
Barring that, some instruction on what to do when the GUI doesn't run would probably be useful
<fanquake>
Taking this to the extreme, some obscure missing GUI dependency seems like a great way to compromise non-technical users
<fanquake>
Similar to the "download this .dll to fix your computer" nonsense you see on Windows
<fanquake>
i.e you can download this (definitely not compromised to extract your private keys) libxcb-cursor0.so to fix your bitcoin-qt installation
f321x has joined #bitcoin-core-dev
WizJin_ has quit [Read error: Connection reset by peer]
WizJin_ has joined #bitcoin-core-dev
<sipa>
_aj_: i think an approach like that may work, but what is the advantage of having a separate decaying average for incoming rate, over effectively.using the queue size itself as that metric?
<sipa>
fanquake: that would be nice to address
<_aj_>
sipa: making this up as i go along, but: smooths out bursts more, and keeps the queue size small during sustained load that's above our expected average?
conman has joined #bitcoin-core-dev
aleggg has quit [Ping timeout: 256 seconds]
cman has quit [Remote host closed the connection]
hirish has quit [Ping timeout: 256 seconds]
aleggg has joined #bitcoin-core-dev
hirish has joined #bitcoin-core-dev
jeremyrubin9 has joined #bitcoin-core-dev
Earnest has quit [Ping timeout: 244 seconds]
helo_ has quit [Ping timeout: 244 seconds]
Earnestly has joined #bitcoin-core-dev
zumbi_ has joined #bitcoin-core-dev
saturday- has joined #bitcoin-core-dev
P8tBoy has joined #bitcoin-core-dev
<sipa>
_aj_: right that has a similar effect to having a timeout on each tx, and once hit
<sipa>
always sending it
hernanmarino_ has joined #bitcoin-core-dev
<sipa>
but not as abrupt a change between timed out and not
roconnor_ has joined #bitcoin-core-dev
roconnor has quit [Killed (NickServ (GHOST command used by roconnor_))]
roconnor_ is now known as roconnor
Guyver2 has quit [Remote host closed the connection]
Henry151_ has joined #bitcoin-core-dev
helo has joined #bitcoin-core-dev
hirish has quit [Ping timeout: 256 seconds]
cmirror has quit [Ping timeout: 256 seconds]
jeremyrubin9 has quit [Ping timeout: 256 seconds]
jeremyrubin1 has joined #bitcoin-core-dev
saturday- has quit [Ping timeout: 256 seconds]
WizJin_ has quit [*.net *.split]
robszarka has quit [*.net *.split]
hugohn______ has quit [*.net *.split]
hernanmarino has quit [*.net *.split]
zumbi has quit [*.net *.split]
Henry151 has quit [*.net *.split]
PatBoy has quit [*.net *.split]
Saturday7 has quit [*.net *.split]
BGL has quit [*.net *.split]
jeremyrubin has quit [*.net *.split]
TheRec_ has quit [*.net *.split]
javi404 has quit [*.net *.split]
pablomartin4btc has quit [*.net *.split]
stratospher[m] has quit [*.net *.split]
cotsuka has quit [*.net *.split]
jeremyrubin1 is now known as jeremyrubin
<sipa>
_aj_: what about: {past_avg += 1} on every insertion into the queue, and on every send {to_send = min(queue, max(TARGET_RATE, past_avg * (1 - Q))); past_avg = (past_avg - to_send) * Q}, where Q = exp(-TIMEOUT/AVG_SEND_INTERVAL)
f321x has quit [Remote host closed the connection]
hirish has joined #bitcoin-core-dev
<sipa>
this way (even with TARGET_RATE=0), every insertion into the queue has an effect on to_send, summed over time, that adds up to 1
Saturday7 has joined #bitcoin-core-dev
TheRec has joined #bitcoin-core-dev
szarka has joined #bitcoin-core-dev
WizJin__ has joined #bitcoin-core-dev
hugohn______ has joined #bitcoin-core-dev
javi404 has joined #bitcoin-core-dev
pablomartin4btc has joined #bitcoin-core-dev
stratospher[m] has joined #bitcoin-core-dev
cotsuka has joined #bitcoin-core-dev
f321x has joined #bitcoin-core-dev
f321x has quit [Remote host closed the connection]
<sipa>
if we're going to add a min(..., 1000), i think we should instead just be dropping things when the queue grows too big - that's preferable to growing our own queue unboundedly
f321x has joined #bitcoin-core-dev
<_aj_>
would dropping from the queue imply `past_avg -= 1` ?
f321x has quit [Remote host closed the connection]
<_aj_>
going above some value would just mean the receiver is dropping them instead of us; that's probably easier implementation-wise :-/
<sipa>
hmm, it should correspond to past_avg -= exp(-time_in_queue / AVG_SEND_INTERVAL)
<sipa>
but we don't know how long something has been in the queue when it gets dropped i guess
f321x has joined #bitcoin-core-dev
<_aj_>
yeah, dropping the max(,1000) seems a lot easier than dropping actual txs
f321x has quit [Remote host closed the connection]
<sipa>
we do have better rate limiting on the receiver side than we did when this mechanism was first introduced
<sipa>
(through the maximum queue size in txrequest)
f321x has joined #bitcoin-core-dev
<_aj_>
sipa: what's TIMEOUT ?
<sipa>
_aj_: the time constant of decaying, so the time during which a fraction (1 - 1/e) of all transactions make it out
<sipa>
could use Q = 1/2^(HALF_LIFE / AVG_SEND_INTERVAL) instead
<_aj_>
hmm, okay, i guess i shouldn't be surprised that a steady-state simulation doesn't tell me much about how it might smooth out bursts
enochazariah has joined #bitcoin-core-dev
andytoshi has quit [Server closed connection]
andytoshi has joined #bitcoin-core-dev
<sipa>
every burst should have an exponentially-decaying impuls response
TheCharlatan has quit [Quit: WeeChat 4.1.1]
TheCharlatan has joined #bitcoin-core-dev
<_aj_>
oh, should Q be scaled based on the actual time between invs? i seem to mostly be seeing max(to_send)=max(q)
<sipa>
that'd be nicer, but i don't see how to guarantee summing up to 1 then
<_aj_>
scale both (1-q) and q?
<_aj_>
ie, (1-rq) and rq
<sipa>
hmm, that may work
<sipa>
my formula is wrong
<sipa>
it should just be past_avg -= to_send
<sipa>
.. and then past_avg just becomes equal to the queue size
<sipa>
i.e., on every send, you send a fraction 1-Q of the remaining queue
twistedline has joined #bitcoin-core-dev
BGL has joined #bitcoin-core-dev
<_aj_>
shouldn't you send a fraction Q of the remaining queue?
<_aj_>
no that's much worse
SpellChecker has quit [Remote host closed the connection]
SpellChecker has joined #bitcoin-core-dev
bugs_ has joined #bitcoin-core-dev
<sipa>
Q is the factor by which the queue shrinks every send
Cory56 has quit [Quit: Client closed]
Cory56 has joined #bitcoin-core-dev
enochazariah has quit [Quit: Client closed]
mbrochh has joined #bitcoin-core-dev
<_aj_>
hmm, Q=0.9 seems like it gives nice behaviour in my sim, though that implies TIMEOUT=0.5s which doesn't seem to make sense to me
<sipa>
with R the target rate (tx/s), and H the half-life time (s)
<sipa>
and t the time since last send
Cory56 has quit [Quit: Client closed]
Cory56 has joined #bitcoin-core-dev
<_aj_>
sipa: seemed okayish even without that i think
Christoph_ has quit [Quit: Christoph_]
<sipa>
could use t=AVG_INTERVAL instead of the actual time since send
<sipa>
in either case, i think the average time spent in the queue will be log(2)*H
<sipa>
no, H/log(2)
f321x_ has joined #bitcoin-core-dev
f321x has quit [Remote host closed the connection]
kevkevin has joined #bitcoin-core-dev
Guest21 has joined #bitcoin-core-dev
Guest21 has quit [Client Quit]
f321x_ has quit [Remote host closed the connection]
f321x has joined #bitcoin-core-dev
Guyver2 has left #bitcoin-core-dev [Closing Window]
jonatack has joined #bitcoin-core-dev
szarka has quit [Ping timeout: 256 seconds]
jonatack has quit [Ping timeout: 250 seconds]
jonatack has joined #bitcoin-core-dev
MrHAPPY has joined #bitcoin-core-dev
enochazariah has joined #bitcoin-core-dev
jon_atack has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 256 seconds]
f321x has quit [Ping timeout: 272 seconds]
f321x has joined #bitcoin-core-dev
jonatack has joined #bitcoin-core-dev
jon_atack has quit [Ping timeout: 244 seconds]
dzxzg has joined #bitcoin-core-dev
jon_atack has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 256 seconds]
enochazariah has quit [Ping timeout: 250 seconds]
f321x has quit [Remote host closed the connection]
f321x has joined #bitcoin-core-dev
mbrochh has quit [Quit: Connection closed for inactivity]
jon_atack has quit [Ping timeout: 258 seconds]
jonatack has joined #bitcoin-core-dev
Saturday7 has quit [Ping timeout: 256 seconds]
jon_atack has joined #bitcoin-core-dev
Saturday7 has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 244 seconds]
Cory56 has quit [Quit: Client closed]
Cory56 has joined #bitcoin-core-dev
jon_atack has quit [Ping timeout: 256 seconds]
jonatack has joined #bitcoin-core-dev
f321x has quit [Quit: f321x]
Cory56 has quit [Quit: Client closed]
Cory56 has joined #bitcoin-core-dev
TallTim_ has joined #bitcoin-core-dev
jon_atack has joined #bitcoin-core-dev
phantomcircuit has quit [Remote host closed the connection]
phantomcircuit has joined #bitcoin-core-dev
dzxzg has quit [Quit: Konversation terminated!]
jonatack has quit [Ping timeout: 258 seconds]
kanzure has quit [Ping timeout: 245 seconds]
gnusha_ has quit [Ping timeout: 265 seconds]
jrayhawk has quit [Ping timeout: 245 seconds]
tarotfied has quit [Ping timeout: 248 seconds]
TallTim has quit [Ping timeout: 248 seconds]
gnusha has joined #bitcoin-core-dev
kanzure has joined #bitcoin-core-dev
jrayhawk has joined #bitcoin-core-dev
tarotfied has joined #bitcoin-core-dev
l0rinc has quit [Quit: l0rinc]
Cory46 has joined #bitcoin-core-dev
Cory56 has quit [Ping timeout: 250 seconds]
abubakarsadiq has quit [Quit: Connection closed for inactivity]
Cory86 has joined #bitcoin-core-dev
l0rinc has joined #bitcoin-core-dev
Cory46 has quit [Ping timeout: 250 seconds]
Cory48 has joined #bitcoin-core-dev
dzxzg has joined #bitcoin-core-dev
Cory86 has quit [Ping timeout: 250 seconds]
jon_atack has quit [Ping timeout: 248 seconds]
jonatack has joined #bitcoin-core-dev
jon_atack has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 256 seconds]
Cory1 has joined #bitcoin-core-dev
Cory48 has quit [Ping timeout: 250 seconds]
saturday- has joined #bitcoin-core-dev
Saturday7 has quit [Ping timeout: 256 seconds]
<dzxzg>
_aj_: sorry, "so bad" was a so bad choice of words and not correct, all I meant was that the current formula struggles struggles to reduce the queue size when transaction rates are above 7 tx/s
<dzxzg>
and when that behavior is sustained without things calming down
Talkless has joined #bitcoin-core-dev
<dzxzg>
I still kind of share the above question asked by _aj_ of why not just always send as much as we can? I'm not sure I understand this answer "to smooth out bursts of low-fee txs so as to avoid wasting the network's validation and relay resources" Isn't the mempool size limit / minrelayfee the mechanism used to prevent this kind of DoS, what is so bad about an attacker being able to get you to send big INV's to your peers?
<sipa>
the current formula roughly corresponds to this new exponential one, but with H=138*AVG_INTERVAL
<sipa>
which is a pretty long half life
<sipa>
dzxzg: i think the idea is that when there are bursts, and we expect (part of) the network to not keep up, we slow things down a bit, to give better transactions that arrive *later* to jump the queue
<sipa>
+a chance
enochazariah has joined #bitcoin-core-dev
l0rinc has quit [Ping timeout: 256 seconds]
Cory48 has joined #bitcoin-core-dev
l0rinc has joined #bitcoin-core-dev
Cory1 has quit [Ping timeout: 250 seconds]
enochazariah has quit [Quit: Client closed]
enochazariah has joined #bitcoin-core-dev
_durandal has quit [Ping timeout: 258 seconds]
jespada has joined #bitcoin-core-dev
jespada has quit [Client Quit]
<dzxzg>
sipa: OK, just reiterating: since we INV in sorted order of `CompareDepthAndScore()`, limiting INV sizes gives the best transactions a chance to skip to the front at every INV interval; this isn't really about the big INV being directly bad, but about the fact that transactions are requested in the order which they are announced, sending big INV's would create large transaction request queues that a high fee transaction that arrives one INV interval
<dzxzg>
after a burst will have to wait behind.
<sipa>
dzxzg: right!
<sipa>
(and CompareDepthAndScore will become much better post-clustermempool too, because it can sort by chunk feerate rather than individual feerate)
jespada has joined #bitcoin-core-dev
_durandal has joined #bitcoin-core-dev
<instagibbs>
Soon(TM)
saturday- has quit [Ping timeout: 256 seconds]
Saturday7 has joined #bitcoin-core-dev
Talkless has quit [*.net *.split]
MrHAPPY has quit [*.net *.split]
BGL has quit [*.net *.split]
TheRec has quit [*.net *.split]
WizJin__ has quit [*.net *.split]
hugohn______ has quit [*.net *.split]
javi404 has quit [*.net *.split]
pablomartin4btc has quit [*.net *.split]
stratospher[m] has quit [*.net *.split]
cotsuka has quit [*.net *.split]
TheRec_ has joined #bitcoin-core-dev
WizJin__ has joined #bitcoin-core-dev
hugohn______ has joined #bitcoin-core-dev
javi404 has joined #bitcoin-core-dev
pablomartin4btc has joined #bitcoin-core-dev
stratospher[m] has joined #bitcoin-core-dev
cotsuka has joined #bitcoin-core-dev
jeremyrubin3 has joined #bitcoin-core-dev
jon_atack has quit [Ping timeout: 256 seconds]
jrayhawk has quit [Ping timeout: 256 seconds]
zumbi_ has quit [Ping timeout: 256 seconds]
MrHAPPY has joined #bitcoin-core-dev
jrayhawk has joined #bitcoin-core-dev
jonatack has joined #bitcoin-core-dev
_durandal has quit [Ping timeout: 256 seconds]
jeremyrubin has quit [Ping timeout: 256 seconds]
zumbi has joined #bitcoin-core-dev
jeremyrubin3 is now known as jeremyrubin
_durandal has joined #bitcoin-core-dev
<sipa>
instagibbs: any release now
jon_atack has joined #bitcoin-core-dev
jespada has quit [Ping timeout: 244 seconds]
jerryf has joined #bitcoin-core-dev
jerryf_ has quit [Ping timeout: 272 seconds]
thoragh has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 256 seconds]
<dzxzg>
sipa: Thanks, that makes sense, and for the other half: we care about the queue growing because it's a memory expense and the CPU expense of sorting the entire queue for each peer every INV, would it be cheaper to avoid the queue entirely and just iterate through the mempool in sorted order, skipping the tx'es that match m_tx_inventory_known_filter?
<instagibbs>
ignoring runtime/etc you'd start advertising old txns to new peers which would be very different behavior
<sipa>
dzxzg: i can't imagine that's faster unless you're close to having the entire mempool in the queue
l0rinc has quit [Quit: l0rinc]
l0rinc has joined #bitcoin-core-dev
jonatack has joined #bitcoin-core-dev
jon_atack has quit [Ping timeout: 256 seconds]
darosior3 has joined #bitcoin-core-dev
darosior has quit [Read error: Connection reset by peer]
darosior3 is now known as darosior
thoragh has quit [Ping timeout: 255 seconds]
kevkevin_ has joined #bitcoin-core-dev
kevkevin has quit [Ping timeout: 256 seconds]
enochazariah has quit [Ping timeout: 250 seconds]
magellan has quit [Quit: WeeChat 4.7.1]
thoragh has joined #bitcoin-core-dev
l0rinc has quit [Quit: l0rinc]
ghost43_ has joined #bitcoin-core-dev
ghost43 has quit [Remote host closed the connection]
BGL has joined #bitcoin-core-dev
PaperSword has joined #bitcoin-core-dev
l0rinc has joined #bitcoin-core-dev
jon_atack has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 256 seconds]
jesseposner has quit [Server closed connection]
jesseposner has joined #bitcoin-core-dev
l0rinc has quit [Quit: l0rinc]
l0rinc has joined #bitcoin-core-dev
jonatack has joined #bitcoin-core-dev
jon_atack has quit [Ping timeout: 244 seconds]
jon_atack has joined #bitcoin-core-dev
bugs_ has quit [Quit: Leaving]
Christoph_ has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 256 seconds]
Ademan has quit [Quit: Reconnecting]
Ademan_ has joined #bitcoin-core-dev
jerryf_ has joined #bitcoin-core-dev
jerryf has quit [Ping timeout: 272 seconds]
Christoph_ has quit [Quit: Christoph_]
jerryf has joined #bitcoin-core-dev
jerryf_ has quit [Remote host closed the connection]