<Murch>
<BlueMatt[m]> "is there a case where bitcoin..." <- Yes, generally the fee estimation can be unreliable if you were offline. We use only transactions that were in the mempool and then got included in a block, so I think if you e.g. went offline a bit before the mempool got cleared out, then started it later it would reload from the mempool file on startup, see that all these transactions have gotten confirmed, including ones that the node has just
<Murch>
received and then you have no new data after that. I'd imagine that might lead to underestimation.
<Murch>
Also, if the short-term data is entry it would fall back to slightly older data, up to the last couple days or so
<Murch>
I'm not sure whether there is a mechanism in place to ensure it does not provide estimates if it has too little data
<Murch>
It might fall back to the default fee in that case
<sipa>
Yeah, there is a fallback fee setting, IIRC
<Murch>
Anyway, it probably takes at least one blocks to be found for the fee estimation to have local information
<sipa>
IIRC you need at least 2x as many blocks as the window you're aiming for to get a reliable estimate.
aielima has quit [Quit: Ciao]
dberkelmans has joined #bitcoin-core-dev
dberkelmans has quit [Ping timeout: 245 seconds]
AaronvanW has quit [Ping timeout: 246 seconds]
gossie has quit [Quit: = "bye bye"]
AaronvanW has joined #bitcoin-core-dev
AaronvanW has quit [Ping timeout: 240 seconds]
flooded has quit [Ping timeout: 240 seconds]
AaronvanW has joined #bitcoin-core-dev
<sipa>
Though, it shouldn't return 1 sat/vb when not enough data is available...
achow101 has quit [Quit: No Ping reply in 180 seconds.]
achow101 has joined #bitcoin-core-dev
<willcl_ark>
glozow / Pieter ref discussion on #19109 (which I can't reply to), just noting here that enabling NODE_BLOOM will permit our node to send peers a response to a `mempool` msg -- it doesn't require additional net permissions on the peer. (h/t b10c)
stick has quit [Quit: Connection closed for inactivity]
<sipa>
@glozow That's crazy... do we know why Umbrel and Raspiblitz are setting peerbloomfilters?
<sipa>
If we do want to keep support for that, I think it's reasonable to indeed exclude non-yet-announced transactions from the BIP35 response.
<glozow>
sipa: I agree 😭. My best guess is it’s for bisq
<sipa>
BIP37 (or BIP111 now...) support should really be enabled just for peers trusted not to DoS.
<glozow>
Yes, unfortunately it seems people do not know this
<glozow>
I think we could open issues to umbrel and raspiblitz to discourage it and maybe put louder warnings, but for now exclude not recently announced
<sipa>
That makes sense.
<darosior>
Murch, BlueMatt[m]: but there is an exponential decay on data points. If all the data we've got is from too long ago we should (and i'm pretty sure do) fail to provide any estimate.
brunoerg has joined #bitcoin-core-dev
<darosior>
About the BIP37 issue i was stunt too so i've asked on Twitter yesterday why do those node-in-a-box set `peerbloomfilters` and a Raspiblitz contributor said it was indeed for Bisq support (as glozow suggested). https://twitter.com/darosior/status/1653037196704727046
<darosior>
It's even crazier when you think they usually come bundled with a Lightning node...
<sipa>
If so, shouldn't it suffice to provide it to 127.0.0.1 ?
flooded has joined #bitcoin-core-dev
<darosior>
I think they provide it through Tor because they use it to remotely connect to their node at home
flooded is now known as _flood
<sipa>
... do they keep the onion address secret?
brunoerg has quit [Ping timeout: 246 seconds]
<darosior>
I don't know but i don't see why they'd share it. It was also an argument from the Raspiblitz contributor: that they only run the bitcoind and lightningd through Tor so it's harder to correlate them... But hey it sounds like a pretty weak protection
<emzy>
FWIW I run 4 bitcoin core public nodes for Bisq, with peerbloomfilters=1. They are very busy but work fine.
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 248 seconds]
brunoerg has joined #bitcoin-core-dev
<darosior>
emzy: could Bisq technically switch to using something like block filters instead?
brunoerg has quit [Ping timeout: 260 seconds]
<furszy>
most likely it's not bisq issue, it's bitcoinj not supporting client side filtering.
<furszy>
will check what they have there, been a while since I worked with it.
<furszy>
ok yep, bitcoinj master doesn't have the node_compact_filters service flag.
<bitcoin-git>
[bitcoin] sipa opened pull request #27552: Add support for "partial" fuzzers that indicate usefulness (master...202305_partial_fuzzers) https://github.com/bitcoin/bitcoin/pull/27552
<furszy>
maybe Bisq, and others, could switch to use core instead of bitcoinj if we provide opt-in client side filtering capabilities.
<furszy>
which.. shilling a bit, is basically what I have been implementing the past months (it’s not fully ready but, short summary: I have a PoC running with p2p, RPC and wallet support to request filters on-demand and automatically, track in-flight and stale requests and request matching blocks + update the wallet state).
<furszy>
this was the reason behind #26966, #27006 and #27469..
<furszy>
so.. will try to get in-touch with Bisq and see what they have planned. Even if this doesn’t get into core soon (or never if there is no consensus), it’s a good use case for them and for us to get rid-off bip35 and bip37 net support.
<gribble>
https://github.com/bitcoin/bitcoin/issues/27469 | wallet: improve IBD sync time by skipping block scanning prior birth time by furszy · Pull Request #27469 · bitcoin/bitcoin · GitHub
<sdaftuar>
jamesob6: yeah i understand the desire to not special-case the download logic. but i was hoping we can avoid parametrizing data in net_processing based on Chainstate -- it seems like most of the complexity in block download is having to deal with the case that we might need to download a chain that forks from our own chain.
<sdaftuar>
jamesob6: so i was thinking that if the background chain download logic really is dead-simple, maybe we can just introduce logic that does that simple thing. i'll play with it and see what i get
Guyver2 has left #bitcoin-core-dev [Closing Window]
<josie>
furszy: sounds cool! so this would be bitcoin core node which asks for blocks on demand based on the wallet / rpc commands?
brunoerg has joined #bitcoin-core-dev
<furszy>
yeah. There are two modes: manual and automatic. The manual mode moves the sync responsibility to any external software (RPC commands to request filters, get the wallet needle set, check if filter matches, request the block, etc). And a automatic mode that follows bip157 client side specs.
<josie>
furszy: id be interested in testing/playing around with this if you have a branch I can compile and run
<furszy>
cool josie :). I’m not that far from sharing it publicly, being pushing to reach to a point where I'm happy enough with the sources first (have a good number of pending to do). Mainly to not share something that I know that will be changing soon and get a better first review/testing experience.
<furszy>
Thus why have been decoupling stuff from that branch into smaller PRs, like the ones shared above, so those can be reviewed in parallel and this branch doesn't become a big scary monster.
<josie>
furszy: cool, looking forward to it!
<furszy>
<3
b_101 has quit [Ping timeout: 256 seconds]
jonatack1 has joined #bitcoin-core-dev
AaronvanW has quit [Remote host closed the connection]
jon_atack has quit [Ping timeout: 264 seconds]
AaronvanW has joined #bitcoin-core-dev
vysn has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] hebasto opened pull request #27554: test: Treat `bitcoin-wallet` binary in the same way as others (master...230502-toolwallet) https://github.com/bitcoin/bitcoin/pull/27554
<instagibbs>
re:estimatesmartfee, if it's degraded, it shouldn't return garbage, but an error, right?
<instagibbs>
(or a fallback fee if enabled, which isn't on mainnet?)
<Murch>
I think if you were just offline for a bit, and the mempool cleared after shut down, the transactions that get loaded with the mempool and the catching up to the chaintip might be the only data to estimate the short-window on
<Murch>
1 ṩ/vB seems odd though, because it should use the maximum from the short-term, mid-term, and long-term bucket
<achow101>
there's a fee_estimate.dat file, so presumably we are loading that with possibly outdated data on start
<instagibbs>
Murch depends on mode, not sure what mode they're using
<dergoegge>
Afaiu this is what they do now instead of picking inputs at random from the corpus, to spend more time on "interesting" inputs
<sipa>
Ah yes, I found that too after looking up what the log message about "entropic power schedule" was about.
<BlueMatt[m]>
<sipa> "Yeah, there is a fallback fee..." <- That’s just for wallet, though, no? The estimator itself should never use that.
<instagibbs>
the estimator is supposed to barf if it's not confident
<BlueMatt[m]>
<Murch> "Yes, generally the fee estimatio..." <- Right, but it won’t invent things out of whole cloth, which would be required to explain what I saw.
<BlueMatt[m]>
instagibbs: Right that’s what the code I read said, I was mostly checking that (a) I read it right and (b) there’s not some other known cases where it returns 1s/van
<BlueMatt[m]>
Vb
nintendo1889 has quit [Quit: Connection closed for inactivity]
<instagibbs>
Only known case I know of is sometimes mempool fills up(and trims) faster than estimator can catch up, so mempool minfee isn't reflected in it
<darosior>
re fee estimation. "there's a fee_estimate.dat file, so presumably we are loading that with possibly outdated data on start" -> Yes but we do store the height in this file iirc. And if it's too far from our current best height we won't serve outdated data.
<fjahr>
I am not sure if this is related to Matt’s issue because we are talking about very different time frames but I just synced a node that wasn’t running since early last month, so it had to catch up about 4 weeks. The mempool was cleared and as it was syncing I requested “estimatesmartfee 2” multiple times because I saw the conversation here. The suggested feerate fluctuated between 0.00006167 and 0.00022090. I don’t
<fjahr>
see how the node would be able to make reliable predictions if it hasn’t seen the blocks of the last few days and nothing in the mempool. Does anyone know on what basis these values are fluctuating in this case?
<BlueMatt[m]>
instagibbs: Squinting there it shouldn’t happen on mainnet though I think?
<instagibbs>
I've never seen it, though I don't restart old nodes a lot and use them
<BlueMatt[m]>
At least the counterparty in one case claimed their node had crashed only a few hours before
<BlueMatt[m]>
Dunno how often the estimates are written to disk during normal operation
<BlueMatt[m]>
So unclear how old that file could have been
<darosior>
BlueMatt[m]: only at shutdown
<BlueMatt[m]>
Oh that’s bad
<instagibbs>
ah, crash
<BlueMatt[m]>
So maybe that was the bug then - restarted with estimates from a month ago or something stupid
<instagibbs>
seems like something we could flush every 24 hours? not a huge file
<BlueMatt[m]>
Yea that plus reject if the estimates haven’t caught up with now
<BlueMatt[m]>
Or otherwise are that old
<BlueMatt[m]>
Or even persist once an hour
<darosior>
Then how comes it would serve old data? Maybe it wasn't synced yet and didn't know it was outdated?
<BlueMatt[m]>
I’d understood fjahr s point to be that it will happily serve stale data
<BlueMatt[m]>
As long as it has data
<darosior>
Yeah we should definitely not serve fee estimation if we are catching up
<fjahr>
instagibbs: hm, possible, I thought I kind of understood that part (narrator: "he didn't") but there is nothing named pkginfo or similar in the generic part. Most likely I would guess the graphics haven't evolved with the text.
AaronvanW has quit [Remote host closed the connection]
<fjahr>
But yeah, ancpkginfo is something different and I guess plain pkginfo has been removed at some point (?)
berndj has joined #bitcoin-core-dev
jamesob6 has joined #bitcoin-core-dev
jamesob4 has joined #bitcoin-core-dev
<jamesob6>
sdaftuar: great! anything that minimizes risk is +1 from me
<instagibbs>
fjahr it's in the "generic" portion imagery, otherwise not defined
<instagibbs>
(i think)
b_101 has joined #bitcoin-core-dev
<fjahr>
yepp, that's what I see too, I guess glozow will have to solve that mystery :)
AaronvanW has joined #bitcoin-core-dev
flooded has joined #bitcoin-core-dev
b_101 has quit [Ping timeout: 264 seconds]
_flood has quit [Ping timeout: 268 seconds]
Talkless has joined #bitcoin-core-dev
b_101 has joined #bitcoin-core-dev
b_101 has quit [Ping timeout: 268 seconds]
b_101 has joined #bitcoin-core-dev
b_101 has quit [Ping timeout: 240 seconds]
aielima has joined #bitcoin-core-dev
<achow101>
24.1rc2 and 25.0rc1 bins are uploaded
AaronvanW has quit [Remote host closed the connection]
b_101 has joined #bitcoin-core-dev
b_101 has quit [Ping timeout: 246 seconds]
Guest55 has joined #bitcoin-core-dev
Guest55 has quit [Client Quit]
<bitcoin-git>
[bitcoin] furszy opened pull request #27556: wallet: fix deadlock in bdb read write operation (master...2023_wallet_db_deadlock) https://github.com/bitcoin/bitcoin/pull/27556
AaronvanW has joined #bitcoin-core-dev
b_101 has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] pinheadmz opened pull request #27557: net: call getaddrinfo() in detachable thread to prevent stalling (master...async-getaddrinfo) https://github.com/bitcoin/bitcoin/pull/27557
<bitcoin-git>
[bitcoin] pinheadmz closed pull request #27505: net: use interruptible async getaddrinfo wrapper from libevent for DNS (master...dns-libevent) https://github.com/bitcoin/bitcoin/pull/27505
AaronvanW has quit [Ping timeout: 240 seconds]
<jamesob6>
anyone ever have a hard time reproducing threadsanitizer errors locally?
jamesob6 is now known as jamesob
AaronvanW has joined #bitcoin-core-dev
test_ has joined #bitcoin-core-dev
flooded has quit [Ping timeout: 240 seconds]
<fjahr>
FYI: For those that were testing my gitlab instance, Mike and me are making some changes as we discuss the topic with them so it's possible that the instance will not be available at some point in the coming days due to new setup or that your account changes.
<fjahr>
DM me if there is something you want to test in the meantime
Talkless has quit [Quit: Konversation terminated!]
flooded has joined #bitcoin-core-dev
test_ has quit [Ping timeout: 252 seconds]
kch has joined #bitcoin-core-dev
b_101 has quit [Ping timeout: 240 seconds]
kch has quit [Quit: Lost terminal]
vysn has quit [Remote host closed the connection]
brunoerg has quit [Remote host closed the connection]
brunoerg has joined #bitcoin-core-dev
vysn has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 240 seconds]
bugs_ has quit [Quit: Leaving]
aielima has quit [Quit: Ciao]
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 248 seconds]
brunoerg has joined #bitcoin-core-dev
AaronvanW has quit [Quit: Leaving...]
brunoerg has quit [Ping timeout: 248 seconds]
brunoerg has joined #bitcoin-core-dev
b_101 has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] 0xB10C opened pull request #27559: doc: clarify processing of mempool-msgs when NODE_BLOOM (master...2023-05-clarify-mempool-bloom) https://github.com/bitcoin/bitcoin/pull/27559
brunoerg has quit [Ping timeout: 240 seconds]
brunoerg has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] furszy closed pull request #26644: wallet: bugfix, 'wallet_load_ckey' unit test fails with bdb (master...2022_walletdb_fix_bdb_deadlock) https://github.com/bitcoin/bitcoin/pull/26644