< sipa>
meshcollider: if we do it for one, we must do it for both
< sipa>
the same reasoning applies
< sipa>
except that decoderawtransaction is somewhat less likely to have impact
< wallet42>
sipa: what bech32hrp do you have in mind for these ?
< sipa>
wallet42: i'm working on searching for a 12-character checksum with much stronger properties
< wallet42>
how did you hack space-time? you have more that 24 hours in a day?
< sipa>
wallet42: by 'working' i mean i have some computers running analysis
< sipa>
where 'some' means 176 cpu cores ;)
< wallet42>
i meant that you have an incredible work pace
< sipa>
heh, thanks :)
< meshcollider>
he's right haha you seem to be working on everything
< luke-jr>
lol
< wallet42>
I was thinking about an easy to implement yet better fee algo for wallet developers. The current situation is bad because many wallets (most that i used) want good user experience and thus chose a fee for them that puts it at the top of the mempool priority queue. If the mempool is bigger than 1 MB, that means they are constantly outcompeting each other.
< wallet42>
My simple fix would be (by default) use an opt-in RBF. Look at the mean feerate (sat/byte) in the mempool. Substract 1 sat/byte. Use that fee. If the user then complaints, offer him to increase fee via RBF after 1 hour no confirmation.
< wallet42>
Now there is a drag down instead of drag up in terms of average/mean feerate.
< sipa>
wallet42: i don't think average necessarily means anything
< sipa>
anyone can create a near infinite amount of transactions of low fee
< bitcoin-git>
[bitcoin] AkioNak opened pull request #11173: RPC: Fix currency unit string in the help text (master...use_currency_unit_constant) https://github.com/bitcoin/bitcoin/pull/11173
< webuser322>
np, and the segwit settings faq/guide for miners... the things one think first in the morning ;)
< webuser322>
\quit have a good day core!
< gmaxwell>
mental note: we need release notes on the topup stuff and instructions for rescanning
< kallewoof>
After seeing a bunch of complaints about fee estimation on Bitcoin Core I decided to do some profiling on how well the estimator is doing. My approach may be off, but it seems to be overshooting by a ton.
< kallewoof>
Approach: every 100 txs that go into mempool, do an estimateSmartFee for 1..10 blocks for conservative and the same for non-conservative, and store that. For every new block, iterate over all stored estimations and increment blocks waited, and check if the fee per kb for each tx is greater than the lowest 10 fee transactions in the block. If it is, count it as "would have gone into the block".
< gmaxwell>
right now is not a very useful time to look, because the network just spent several days with 50% hashrate gone, and is likely about to spend several days with it gone again.
< kallewoof>
Feels like something that estimator should take into account. Ideally. As long as it's possible for random miner groups to fork off and make random chains the hash rate is going to be very unstable in the foreseeable future. I guess.
< gmaxwell>
as far as your strategy goes, one problem is that blocks frequently contain transactions which have fees far lower than would be accepted for your own txn.
< gmaxwell>
Because the miner is being paid out of band, invisibly to us.
< kallewoof>
Is it that common?
< gmaxwell>
yes, enough to seriously distort estimates. It really is the case that they're too high now, however... just pointing out the methodology limitations.
< kallewoof>
Wow. TIL
< kallewoof>
FWIW the results I'm seeing after a couple of blocks, even though it may be of limited value as you pointed out:
< wallet42>
Is there a website to track out-of-band transaction? eg. mined tx that has not been seen by any (or at least all of the watching) nodes before?
< sipa>
kallewoof: fee estimation isn't looking at the fees in blocks; it's looking at how long blocks take to confirm
< gmaxwell>
wallet42: no, but your debug log will show you them if you have compact block debugging enabled.
< sipa>
and given that it aims for (afaik?) 85%, i would indeed expect that in the majority of cases it overshoots
< wallet42>
undershoot + rbf!
< kallewoof>
sipa: I'm trying to find a way for the (arguably common) case where a user wants their tx to go into the next couple of blocks for a reasonable fee. Lots of people seem to manually pick a fee and lots of others are baffled by the high fees estimated.
< kallewoof>
So I do estimateSmartFee(1..10) and then keep that around and see when the estimation actually put me in towards the bottom fo the block. Is my reasoning off on that? Aside from what gmaxwell said about miners being paid off-band.
< gmaxwell>
right now if you pick a reasonable fee for 4 blocks based on the mempool it is very likely you will not make it in in 20 blocks... because the hashrate will drop soon, so it may not be as high as you think.
< kallewoof>
gmaxwell: if the hashrate is lower than expected, shouldn't the estimator undershoot?
< gmaxwell>
kallewoof: it would be better if instead you looked at txn it you mempool and saw when they confirm, rather than looking at the block.
< sipa>
kallewoof: right... i'm certainly not saying the current approach is the right one, but trying to see if an estimation algorithm that is designed to overshoot overshoots seems hardly useful :)
< kallewoof>
gmaxwell: but i'm trying to profile estimateSmartFee(); doing what you say seems like a great next step though.
< gmaxwell>
kallewoof: no, undershooting is quite bad when no rbf is used, the conservative one should hopefully almost never understoot.
< kallewoof>
sipa: well, 'overshoot' and 'overshoot by 403247 satoshi/k' are different things, IMO
< gmaxwell>
kallewoof: the setting for confirmation targetst is when is quite sure to confirm by, not most likely to.
< sipa>
kallewoof: fair
< gmaxwell>
kallewoof: so for example, if you predict that fee X will confirm in 5 blocks, you shouldn't think there was an overshoot unless _ALL_ mempool txn with fee X> confirmed by 5 blocks.
< gmaxwell>
(or virtually all, like 95% of them)
< kallewoof>
Got it. What I'm seeing now is that txs with "confirm within 5" are going into the next block rather consistently. But this may be because of what you said about the out of band miner payment. Bleh. Maybe I should take the bottom 10 that I also saw in the mempool. I think I can do that.
< gmaxwell>
well no, right now it really is overshooting. It just doesn't respond fast enouh for capacity changing 100% in hours.
< kallewoof>
Then again, nothing prevents these people from also broadcasting their super-low-fee tx so that approach isn't really ideal either.
< wumpus>
gmaxwell: apparently I indeed misconfigured the time machine, now it looks like I bumped 0.14.99 and 0.15.99 in exactly the same second
< kallewoof>
gmaxwell: I think there are two incompatible approaches here: (1) I want my tx to confirm with as low a fee as possible; I can wait a bit, but obviously not forever; (2) I want my tx to confirm within the next few blocks; I am willing to pay extra for this. Right now everyone is being lumped into (2) when it seems a lot of people are of mindset (1).
< gmaxwell>
kallewoof: you can just look at your mempool, and ask for each fee >=X in the mempool at time t, what percentage of them were confirmed by T+5. Our conservative estimator is designed to try return a number that makes that figure very high like 95%.
< gmaxwell>
trying to get e.g. 1 block however requires looking at the mempool, which is very hard to do safely with an unsupervised approach.
< gmaxwell>
(keep in mind we're limited to use approaches which attackers have limited ability to influence)
< kallewoof>
Does it not look at mempool for >1 conf case? I didn't see any conditionals about that.
< gmaxwell>
e.g. you could look at the mempool right now and see what would get you in... assuming your mempool was current this would work, until there was some softfork or transaction censorship your node didn't know about, and in that case an attacker could make you think 100BTC fees weren't making it in...
< sipa>
it only looks at how long mempool txn take to confirm
< sipa>
it doesn't look at the composition of the mempool itself, nor the composition of blocks itself
< gmaxwell>
unconfirmed txn don't figure into its reasoning because of those influence issues.
< kallewoof>
Would a "use mempool to optimize fee" option in the fee estimator make sense? It would only ever lower the fee and it would use the mempool to figure out where approximately to place the fee to get into the bottom 10%.
< kallewoof>
Maybe it would require or default-on rbf to give you a way out if it gets stuck..
< wumpus>
luke-jr: #7339 seems a mess to me; no one in practice will want to build without libevent support; it will create exactly the same user confusion as #10826 "oh I can build without boost?" except there isn't even the excuse that it will be better in the future
< gmaxwell>
Esp if RBF is in use and your mempool is in sync, bidding directly against it is a great approach. But we haven't figured out how to make it safe (if it's even possible).. though it could be used to only lower the estimates vs the current approach. I think we mostly don't do that now because RBF support is new, and really won't be fully complete until after we've switched the wallet to segwit (
< gmaxwell>
because some of the bumping corner cases are not reasonable to solve in the presence of malleability)
< gmaxwell>
kallewoof: jinx.
< gmaxwell>
kallewoof: yes, I think it would, if it only lowers, and if RBF is in use... and if the bumping is a bit more effective than what we have now.
< gmaxwell>
though we may also need to complete mempool sync to get useful enough data shortly after startup. :( oh well it would still be useful if it wasn't available all the time.
< wumpus>
luke-jr: and it does add yet another configuration combination to test
< kallewoof>
Okay! Thanks for the feedback. :) I will look into that for sure.
< bitcoin-git>
bitcoin/master e254830 Andrew Chow: Make tabs toolbar no longer have a context menu...
< bitcoin-git>
bitcoin/master 5b8af7b Wladimir J. van der Laan: Merge #11169: [GUI] Make tabs toolbar no longer have a context menu...
< bitcoin-git>
[bitcoin] laanwj closed pull request #11169: [GUI] Make tabs toolbar no longer have a context menu (master...rm-hide-tabs) https://github.com/bitcoin/bitcoin/pull/11169
< wumpus>
anyhow I"m open to ideas, I'll not be on IRC a lot today and tomorrow (packing etc) but if we want to tag something we need to do it soon
< gmaxwell>
well, morcos fix needs to be merged in 0.15 branch. it applied cleanly for me and thats what I was testing on (0.15+it)
< wumpus>
yes, but for 0.15.0 or 0.15.1?
< wumpus>
obviously it needs to go into some 0.15 release
< sipa>
#11145 is a crashing bugfix
< sipa>
i think it needs to be in 0.15.0
< wumpus>
ok
< wumpus>
so merge it into 0.15 and do a rc3 now?
< sipa>
sgtm
< gmaxwell>
ACK..
< gmaxwell>
This will put the release on what day most likely?
< gmaxwell>
(asking for my slides, I'm doing a presentation on 0.15 tomorrow)
< wumpus>
sept 14/15
< wumpus>
given that we don't need to do another rc, of course
< wumpus>
I never expected this to slip so far
< wumpus>
let's do 0.15 quickly, they say, we need it soon,...
< gmaxwell>
this slip is due to a regression bug at least, we would have made the announced schedule but for it.
< gmaxwell>
so I think thats pretty good.
< wumpus>
it's too early to say
< gmaxwell>
it's not like sse4 sha2 or something we did at the last minute is biting us, and the change that introduced this bug itself fixed an important issue.
< kallewoof>
Just to clarify, there's no code that prioritizes transactions based on lifetime anymore, right? I.e. the higher fee from a longer confirmation window is simply based on probability
< kallewoof>
Er, lower fee I meant
< wumpus>
yes. Though: the 'priority' code didn't prioritize based on transaction lifetime, but lifetime of the input utxos
< bitcoin-git>
bitcoin/master de9a1db Russell Yanofsky: Acquire cs_main lock before cs_wallet during wallet initialization...
< bitcoin-git>
bitcoin/master df91e11 Wladimir J. van der Laan: Merge #11126: Acquire cs_main lock before cs_wallet during wallet initialization...
< bitcoin-git>
[bitcoin] laanwj closed pull request #11126: Acquire cs_main lock before cs_wallet during wallet initialization (master...pr/loadlock2) https://github.com/bitcoin/bitcoin/pull/11126
< bitcoin-git>
[bitcoin] laanwj opened pull request #11176: build: Rename --enable-experimental-asm to --enable-asm and enable by default (master...2017_08_non_experimental_asm) https://github.com/bitcoin/bitcoin/pull/11176
< gmaxwell>
wumpus: ^ thanks, would have done myself but was busy.
< wumpus>
np, I hope I got the build system stuff right
< bitcoin-git>
bitcoin/master 5f850b0 Karl-Johan Alm: [bench] Include ms/blk stats in Connect* benchmarks.
< bitcoin-git>
bitcoin/master a473eff Karl-Johan Alm: [bench] Replace 0.00(000)1 with MICRO/MILLI #defines in validation.cpp.
< bitcoin-git>
bitcoin/master f088a1b Wladimir J. van der Laan: Merge #10303: [bench] Include ms/blk stats in Connect* benchmarks....
< bitcoin-git>
[bitcoin] laanwj closed pull request #10303: [bench] Include ms/blk stats in Connect* benchmarks. (master...bench-add-per-block-stats) https://github.com/bitcoin/bitcoin/pull/10303
< bitcoin-git>
[bitcoin] rawodb opened pull request #11177: Support for SegWit Addresses in RPC calls (master...pr/rpc_getsegwitaddresses) https://github.com/bitcoin/bitcoin/pull/11177
< kallewoof>
Accidentally posted in #bitcoin-dev instead of here, but I got some initial mempool optimizing code for fee estimation in place that looks promising. Will check back when I have more profiling done.
< kallewoof>
Avg 2953 sat/k/tx overshoot seems useful for sure, compared to 370542 (conservative) or 116949 (non-conservative).
< bitcoin-git>
[bitcoin] MeshCollider opened pull request #11178: Add iswitness parameter to decode- and fundrawtransaction RPCs (master...201708_rawtx_bool) https://github.com/bitcoin/bitcoin/pull/11178
< instagibbs>
kallewoof, is this for mempool-based sniping?
< kallewoof>
instagibbs: not sure what you mean by sniping, but it uses mempool to optimize fees
< kallewoof>
wumpus: I may be mixing things up, but was this not changed in a recent merge (or was it a not-merged-yet PR?): "SHA256 hashing has been optimized for architectures supporting SSE 4 (See PR 10182). SHA256 is around 50% faster on supported hardware, which results in around 5% faster IBD and block validation. In version 0.15, SHA256 hardware optimization is disabled in release builds by default, but
< kallewoof>
can be enabled by using --enable-experimental-asm when building.
< kallewoof>
" (from release notes in 0.15)
< kallewoof>
Oh it's a few lines up. I'll add a note to that PR about the release notes in case it's meant for 0.15
< wumpus>
fanquake: I can't do that
< wumpus>
kallewoof: that release note is correct for 0.15
< wumpus>
after 0.15 we'll enable the asm by default
< promag>
wumpus: will you submit a patch to add the runtime flag?
< kallewoof>
Gotcha
< wumpus>
what runtime flag?
< promag>
to enable/disable sha256 asm
< wumpus>
oh no, I'm not going to do that
< promag>
but do you think is really useful?
< wumpus>
no, it wouldn't be useful for me, my point was that *if* user switching of the sha256 method should be possible it should be at run time
< promag>
ok, agree
< kallewoof>
"This can be useful if running a pruned node, since loading a wallet where the most recent sync is beyond the pruned height results in having to download and revalidate the whole blockchain." << Really? Not just the blocks starting at the wallet's last sync point?
< wumpus>
when there are a lot of different ones it might be useful to be able to switch for benchmarking, on the other hand 'bench_bitcoin' should likely just test them al
< wumpus>
no need for user involvement
< kallewoof>
(^ for multi wallets in QT being synced even tho only one is shown)
< wumpus>
kallewoof: I think 'beyond' is confusing there, isn't 'before' meant?
< promag>
wumpus: I thought the same regarding benchmarking
< wumpus>
kallewoof: if the block at time of the last sync of a wallet was pruned, it re-fetches everything
< kallewoof>
Maybe English is failing me. It sounds to me like it has to redownload everything from genesis if a wallet happens to not have been synced for the last (blocks remaining afte pruning) + 1 blocks.
< wumpus>
that's what it does, yes
< kallewoof>
Oh... kay.
< wumpus>
it would be possible to download only a subset of the blocks, but that never happens
< kallewoof>
Maybe wallet.dat files should have an age :)
< kallewoof>
Btw I've been meaning to mention this. Kalle Alm = Karl-Johan Alm for the release notes. Didn't feel important enough to bring up but I saw someone make a PR to fix names before.
< wumpus>
feel free to submit a PR to fix it
< kallewoof>
Sure thing. Since someone was doing it for a group of people I was sort of wondering if there was a place where this was written down.
< wumpus>
there was an editing phase on the wiki, but now it's been merged back to the branch
< maaku>
I added a new opcode to script/interpreter.cpp that uses data serialization, and on 0.15 (but not prior) I get a compilation error because memory_cleanse can't be found when linking bitcoinconsensus. Is it intentional that this support function is excluded from libbitcoinconsensus?
< sipa>
maaku: i don't see why it should be excluded from libbitcoinconsensus
< maaku>
sipa: hrm. that would require linking against ssl
< sipa>
maaku: ugh, we need to replace that function with something non-openssl then
< sipa>
the openssl implementation is braindead anyway
< bitcoin-git>
[bitcoin] dooglus opened pull request #11184: Use IsMine() to check whether we own the custom change address... (0.15...change_ismine_0.15) https://github.com/bitcoin/bitcoin/pull/11184