< gmaxwell>
should UpdatePreferredDownload be checking ffeeler?
< BlueMatt>
probably?
< BlueMatt>
I'm hugely concerned about current fee estimation algorithms - they have effectively put floors on what they will return, when things an order of magnitude or less fee will easily get confirmed :(
< gmaxwell>
huh?
< BlueMatt>
eg currently, because no one creates txn with things less than 50 sat/byte, no fee estimation will ever return anything less than that
< BlueMatt>
but things with 10 often easily get confirmed after 6 or 10 blocks (now, though that can vary to a full day sometimes)
< BlueMatt>
eg i just tested a tx with 0.1 sat/byte fee, and it had issues propagating, but was even still confirmed in 5 blocks
< BlueMatt>
but every fee estimation anywhere returns >50 sat/byte
< sipa>
0.1 sat/byte? you mean 25 sat for a full transaction?
< sipa>
are you sure you're not off by an order of magnitude or 2?
< BlueMatt>
(ofc its looking at mempool state, which is also kinda shit, but...)
< gmaxwell>
mempool state is great, if you're getting it from a miner.
< TD-Linux>
shouldn't overpaying be easy to verify by running the estimator against historical data?
< BlueMatt>
gmaxwell: well, fair
< gmaxwell>
(thus have no issues with non-standard txn, and no its complete)
< gmaxwell>
er know
< gmaxwell>
part of the challenge is miners that @#$@# set their blocksize to less than the maximum... hard for an estimator to know if the block wasn't full because there wasn't supply or because the size was artifically restricted.
< BlueMatt>
TD-Linux: current algorithms are all of the "look at what we had already seen on the public network get into a block to avoid any trickery by miners" variety....the problem is these algorithms tend to have no way to correct downwards if everyone is using them
< BlueMatt>
gmaxwell: yea, and lots of shit that pays 0.1 sat/byte that gets mined :p
< gmaxwell>
BlueMatt: lots of things that get pushed through miner priority interfaces and pays god knows what gets mined.
< BlueMatt>
yea, that too
< BlueMatt>
I dont have a solution, just noting that things are currently Fucked (tm)
< gmaxwell>
which is why you can't simply use the existance of low fee txn getting mined as a measurement of anything. You need to use things not getting mined as your measurement. :)
< BlueMatt>
lol, yes, thank you, I'm well aware of restrictions, but also noting things like mempools on many public nodes being <1MB and miners who post their mempool having the same thing is pretty strong supporting evidence of things being Fucked
< TD-Linux>
BlueMatt, I don't see why they can't correct downward. they would just be delayed one block
< TD-Linux>
err nevermind I see.
< BlueMatt>
TD-Linux: well /someone/ has to be generating such transactions for that to happen, and in sufficient volume
< sipa>
i nominate BlueMatt to do the low fee tx creation job
< BlueMatt>
lol, i nominate petertodd...doesnt he already?
< gmaxwell>
we might want to consider renaming the commandline arguments for minimum relay fee... so as to throw off settings people set back during the flood attacks before there was mempool limiting that have just been forgotten.
< BlueMatt>
yea, i was wondering about that
< gmaxwell>
petertodd does create low fee txn, IIRC the openstransactions stuff produces 1s/b transactions and just RBFs them until they get mined.
< BlueMatt>
yup
< BlueMatt>
yea, I'm now wondering how much of this comes from stuff failing to relay quickly due to shit like that
< TD-Linux>
does anyone collect public historical mempool data?
< gmaxwell>
what does public mean there?
< TD-Linux>
posted publicly
< sipa>
i sort of do
< gmaxwell>
there are some websites with stats but they're all (?) borken because they're all (?) running the aformentioned adjusted settings.
< petertodd>
gmaxwell: opentimestamps starts with the lowest fee bitcoin core will accept with default settings, and increments it by the minimum fee necessary to relay a replacement each time
< BlueMatt>
or displaying miles of garbage that has been floating around for months :(
< gmaxwell>
BlueMatt: that would be an improvement...
< gmaxwell>
well people aren't using charting interfaces that make sense. Megabytes of mempool is a worthless metric... should be 300 all the time, short of expiration.
< petertodd>
gmaxwell: any miner with a mature mempool significantly greater than 1MB will likely be mining replacement OTS transactions rather than the first ones broadcast, assuming propagation is reasonably reliable anyway
< BlueMatt>
gmaxwell: yea, its no where near that much of the time, however
< BlueMatt>
petertodd: i think the propagation assumption is very weak :/
< petertodd>
BlueMatt: last I checked the logs, usualy the OTS tx that actually gets mined is after multiple attempts, IIRC ~5 or so
< BlueMatt>
petertodd: do you have stats on the propagation prior to that point?
< petertodd>
BlueMatt: point is, after that many attempts, it's likely that at least one will have gotten through
< gmaxwell>
BlueMatt: well if you want to write a PR that renames the minrelayfee option... I'd test and ACK it.
< BlueMatt>
yea, ok, willdo
< petertodd>
BlueMatt: after # of replacements before a OTS tx gets mined is 11
< petertodd>
BlueMatt: *average #
< BlueMatt>
petertodd: yea, I'm curious about propagation
< petertodd>
BlueMatt: in 44s I'll give you a txid...
< petertodd>
BlueMatt: oh, wait, no a bit longer than that... gotta wait for a block
< petertodd>
BlueMatt: after a timestamp is confirmed, it waits a minimum timeout interval, then sends a timestamp tx the next time a block is found; after that every block found results in the fees being bumped
< BlueMatt>
k
< petertodd>
BlueMatt: just checked my logs, looks like 50% of the time where multiple replacements happen, the last replacement to be sent is the one that gets mined
< petertodd>
BlueMatt: so I'd say that's pretty suggestive evidence that about 50% of the hashing power is mining opt-in RBF
< petertodd>
here's a recently sent txid, the start of a new replacement chain: 2adceebdd5149b9f6c27764f7a619e69487e55ac37bee2d6f492067585b23f0e
< BlueMatt>
hmm, or at least a big chunk (eg btcc and bitfury, i assume)
< BlueMatt>
in any case, regarding ignoring opt-in rbf for fee calculation, i suppose it might make sense if we only use it for high-confirmation requests (eg 3/10 confirmations)
< gmaxwell>
well it's really easy to see who does and doesn't... the ones that don't (e.g. viabtc) collect a lot less in fees.
< BlueMatt>
but it certainly will have an impact on our 95% threshold
< morcos>
BlueMatt: take a deep breath! this is actually one thing fee estimatation does pretty well
< morcos>
(oops, didn't catch up all the way on backlog)
< morcos>
the reason we don't get estimates that low is the threshold is set very high and we don't have estimates for greater than 25 blocks
< BlueMatt>
im aware
< BlueMatt>
but that doesnt explain why estimates are currently in the 50s, and not the 10s
< morcos>
it does explain, the estimate is answering the question of i want a 95% chance of being filled in my target
< morcos>
10s will not get you filled withing 25 blocks wiht 95% chance
< BlueMatt>
yes, see later discussion, i believe that is in part due to propagation issues
< morcos>
10 sat very recently has had a 75% of being confirmed in 16 blocks and over a longer time period a 75% chance of being confirmed in 32 blocks
< morcos>
for 95% chance you have to go up to a whole day
< morcos>
thats not due to propagation issues i doubt
< morcos>
i think pretty much everyone would propagate > 10s byte
< morcos>
thats due to blocks being mostly full of stuff that pay more than that
< morcos>
as mentioned, recently it has come down
< BlueMatt>
im not convinced...many blocks being generated now are not full because there arent txn to include if you make 8 outbound connections
< BlueMatt>
on some of my public non-listening nodes they do not have 1MB of txn in their mempool
< BlueMatt>
(and havent all day)
< morcos>
i'd bet you just happen to be looking at lull in transaction backlog
< BlueMatt>
these nodes have been online for months
< morcos>
if it stays like this for a week... estimates will come down
< BlueMatt>
thats possible, but also concerning
< morcos>
yes, i'm saying a few days ago.. your mempool had more expensive txs in it
< BlueMatt>
10 sat very recently has had a 75% of being confirmed in 16 blocks
< BlueMatt>
what is your "very recently"?
< morcos>
checking.. normal fee esitmation has a half life of about 2.5 days
< morcos>
i have a node with short term and long term estimates
< morcos>
the short term estimate has halflife of 6 hours
< morcos>
sigh.. just realized it crashed 2 days ago... so that number was from 2 days ago
< BlueMatt>
ahh, grrr
< morcos>
it is true that the existing algorithm doesn't respond very quickly to changes in conditions
< morcos>
anyway.. i have to run now.. i'll restart that node
< morcos>
and we can discuss in more detail later.. but i have TONS of historical data on this...
< morcos>
the most important thing to actually let people place lower fee txs would be to have a way to say something like i want a 75% change of being confirmed in the next 6 hours
< morcos>
in otherwords be able to decrease confidence and increase target
< morcos>
then you'll get much lower estimates
< BlueMatt>
yes, I super think we should set lower confidence thresholds for people who ask about 20 blocks
< BlueMatt>
i mean if you're asking about 20 blocks I'd hope you understand its +/- 100 at that point
< morcos>
agree entirely
< morcos>
and its easy to do combo stuff
< morcos>
ok well they want 20 blocks lets give them a 50% chance its within 20, but a 90% chance its within 40
< morcos>
i mean you just ask both questions and give the max
< BlueMatt>
yea
< BlueMatt>
i mean that sounds like a uselessly complicated api
< BlueMatt>
(how many people will actually use that?) but maybe
< morcos>
well yeah i'm assuming that the default will be that happens behind the scenes
< morcos>
but that if for some reason you don't like whatever magic core is doing, then you can ask the direct questions
< BlueMatt>
sure, ok
< morcos>
anwyay, lets discuss later...
< morcos>
got to run
< morcos>
BlueMatt: I have another node that has been consistently running that keeps estimates out to 1000 blocks but only at the default decay...
< morcos>
For now it has enough data to accurately report lower fee rates.. but you are right actually that there are very few txs down there.. and so if no one is asking about those less quick confirmations.. (b/c we don't give them a way to)
< morcos>
then no one will places txs at those lower feerates and we may eventually not have any data points
< morcos>
so probably we should provide that opportunity sooner rather than later.. i wrote the code over a year ago.. but it was a bit complicated and i didn't want to worry about tryng to get it reviewed
< morcos>
anyway, here is some data.. # blocks target: 95% threshold (current default) , 75% target (maybe what we could give)
< morcos>
16: 60000 , 4600
< morcos>
32: 40000, 4600
< morcos>
64: 10000, 2500 (can't get lower due to lack of data)
< morcos>
128: 4600, 2500
< morcos>
My guess is things less than 10000 (10 sat/byte) have a problem reaching the very high thresholds b/c there are probably some miners with minrelayfee set at 10
< morcos>
for good measure
< morcos>
8: 65000, 15000
< BlueMatt>
morcos: units???
< gmaxwell>
So I have a question-- would we ever want to have support for trusted feerate information? e.g. to be used when your own estimator doesn't have enough data, or such that it only decreases your estimates?
< BlueMatt>
if we have bumpfee........maybe
< BlueMatt>
i mean its not like all other wallets dont have to....
< luke-jr>
if you mean alert-style signed-by-someone, I think it would need to be outside the reference codebase probably, unless we want to change the policy on such things altogether
< luke-jr>
unless perhaps it'd be signed by the miners themselves
< gmaxwell>
luke-jr: I mean something you could configure, which wouldn't have a default. "Get fee estimates from key XYZ"