< andytoshi> achow101: https://github.com/bitcoin/bitcoin/issues/20347 i'm not crazy :P
< andytoshi> lol what a waste of a sunday
< sipa> sounds like a very useful sunday
< andytoshi> hehe, i did actually learn a lot about core
< achow101> I think you've run into why everyone is afraid of changing coin selection. the logic is too hard to follow
< sipa> it's probably a good reason to justify changes
< sipa> the existing logic is a painful mess, but at least my impression was that it was very unlikely to actually fuck up
< andytoshi> i mean, this is a real corner case, achow and i weren't sure if we could make it trigger without simulating weird fee behavior, and even then
< andytoshi> what's surprising is that it is a "coin selection" bug which is entirely outside of SelectCoins, it's the retry-loop in CreateTransaction
< achow101> unfortunately CreateTransaction is considered part of coin selection behavior because of the loop
< fanquake> wumpus / sipa: may just want to block kengendron251
< sipa> fanquake: going to wait
< sipa> his last message made it sound like a mistake
< wumpus> what did they do? looking at the profile it's either a spammer or a 12 year old :)
< fanquake> Posted a bit of spammy crap, but now it also seems like they've managed to subscribe to the repo, and can't figure out how to unsubscribe.
< aj> deleted comment from 20317 says "I was on this planet well before the computer" so rules out the 12yo theory? :)
< bitcoin-git> [bitcoin] ajtowns opened pull request #20353: configure: Support -fdebug-prefix-map (master...202011-ccache-debug-prefix) https://github.com/bitcoin/bitcoin/pull/20353
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/7e373294a5ae...f70eb51b05de
< bitcoin-git> bitcoin/master faa2f06 MarcoFalke: scripted-diff: [build] Ensure source tarball has leading directory name
< bitcoin-git> bitcoin/master f70eb51 Wladimir J. van der Laan: Merge #20318: build: Ensure source tarball has leading directory name
< bitcoin-git> [bitcoin] laanwj merged pull request #20318: build: Ensure source tarball has leading directory name (master...2011-buildTar) https://github.com/bitcoin/bitcoin/pull/20318
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/f70eb51b05de...663fd92b28c3
< bitcoin-git> bitcoin/master bd93fc9 Andrew Chow: Fix change detection of imported internal descriptors
< bitcoin-git> bitcoin/master 663fd92 Wladimir J. van der Laan: Merge #20266: wallet: fix change detection of imported internal descriptor...
< bitcoin-git> [bitcoin] laanwj merged pull request #20266: wallet: fix change detection of imported internal descriptors (master...fix-desc-change) https://github.com/bitcoin/bitcoin/pull/20266
< bitcoin-git> [bitcoin] MarcoFalke pushed 6 commits to master: https://github.com/bitcoin/bitcoin/compare/663fd92b28c3...05aeeee34f15
< bitcoin-git> bitcoin/master fafce1a MarcoFalke: ci: Move documentation to correct config file
< bitcoin-git> bitcoin/master fa0795f MarcoFalke: ci: Replace TRAVIS_OS_NAME with CI_OS_NAME
< bitcoin-git> bitcoin/master fa8b111 MarcoFalke: ci: Run arm ci config on cirrus
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #20339: ci: Run more ci configs on cirrus (master...2011-moreCirrus) https://github.com/bitcoin/bitcoin/pull/20339
< bitcoin-git> [bitcoin] laanwj pushed 6 commits to master: https://github.com/bitcoin/bitcoin/compare/05aeeee34f15...4fd37d0a104f
< bitcoin-git> bitcoin/master fa762a3 MarcoFalke: test: Remove unused unnamed parameter from block.serialize call
< bitcoin-git> bitcoin/master fa1dea1 MarcoFalke: test: Fix deser issue in create_block
< bitcoin-git> bitcoin/master fac865b MarcoFalke: test: Fix intermittent feature_taproot issue
< bitcoin-git> [bitcoin] laanwj merged pull request #20292: test: Fix intermittent feature_taproot issue (master...2011-testFixes) https://github.com/bitcoin/bitcoin/pull/20292
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #20354: test: Add feature_taproot.py --previous_release (master...2010-testFeatureTaprootPreviousVersion) https://github.com/bitcoin/bitcoin/pull/20354
< bitcoin-git> [bitcoin] practicalswift opened pull request #20355: fuzz: Check for addrv1 compatibility before using addrv1 serializer/deserializer on CSubNet (master...fix-sub_net_deserialize) https://github.com/bitcoin/bitcoin/pull/20355
< bitcoin-git> [bitcoin] laanwj pushed 4 commits to master: https://github.com/bitcoin/bitcoin/compare/4fd37d0a104f...79a3b59cc706
< bitcoin-git> bitcoin/master ba8997f Jon Atack: net: update GetNetworkName() with all enum Network cases
< bitcoin-git> bitcoin/master 9a75e1e Jon Atack: rpc: update GetNetworksInfo() to not return unsupported networks
< bitcoin-git> bitcoin/master 7b5bd31 Jon Atack: test: add getnetworkinfo network name regression tests
< bitcoin-git> [bitcoin] laanwj merged pull request #20120: net, rpc, test, bugfix: update GetNetworkName, GetNetworksInfo, regression tests (master...fix-getnetworkinfo-empty-networks) https://github.com/bitcoin/bitcoin/pull/20120
< real_or_random> can we reopen https://github.com/bitcoin/bitcoin/issues/17802 ? I think this needs action before the end of the year.
< sipa> MarcoFalke: ping ^
< az0re> Hey all, just want to point out the totally broken fee estimator
< az0re> !mempool
< gribble> Mempool info: [txcount: 3127, vsize (MB): 7.3364238739, totalfee (BTC): 1.01938529] | Next block 0: [max fee: 12773.8128079, min fee: 1.00305810398] | Next block 1: [max fee: 1.00303951368, min fee: 1.0013713851] | Next block 2: [max fee: 1.00137059849, min fee: 1.001347601]
< az0re> My current mempool histogram shows:
< az0re> [[236, 101197], [205, 109178], [15, 123943], [8, 135543], [1, 7467948]]
< az0re> This is while the mempool has deflated from ~40MB 12 hours ago to ~8MB now
< sipa> who runs gribble these days?
< sipa> nanotube?
< az0re> Not sure
< az0re> Here's what estimatesmartfee gives me:
< az0re> user@host:~$ ./bitcoin-0.20.1/bin/bitcoin-cli -rpcuser=user --rpcpassword=... estimatesmartfee 1 CONSERVATIVE
< az0re> {
< az0re> "feerate": 0.00231653,
< az0re> "blocks": 2
< az0re> }
< az0re> user@host:~$ ./bitcoin-0.20.1/bin/bitcoin-cli -rpcuser=user --rpcpassword=... estimatesmartfee 10 CONSERVATIVE
< az0re> {
< az0re> "feerate": 0.00231653,
< az0re> "blocks": 10
< az0re> }
< az0re> user@host:~$ ./bitcoin-0.20.1/bin/bitcoin-cli -rpcuser=user --rpcpassword=... estimatesmartfee 10 ECONOMICAL
< az0re> {
< az0re> "feerate": 0.00013714,
< az0re> "blocks": 10
< az0re> }
< az0re> user@host:~$ ./bitcoin-0.20.1/bin/bitcoin-cli -rpcuser=user --rpcpassword=... estimatesmartfee 1 ECONOMICAL
< az0re> {
< az0re> "feerate": 0.00199988,
< az0re> "blocks": 2
< az0re> }
< az0re> both ECONOMICAL and CONSERVATIVE suggest *way* too high fees, and I suspect that estimatesmartfee is creating its own reality here: It's suggesting high fees, and so the mempool gets a bunch of really high fee txes, and so the estimator keeps suggesting really high fees
< az0re> However, txes are clearing at way, way lower feerates
< sipa> az0re: that's kind of expected; the estimator does not look at the mempool, but only at the rates of which feerates confirm
< az0re> And if all installed estimators suddenly cut their suggested feerates 10x I suspect nothing would change but people saving a TON on tx fees
< sipa> so it's inherently delayed
< az0re> sipa: Looking at the rates of which feerates confirm should *also* result in a much lower suggested fee
< az0re> In the last 12h I've cleared--in one block--some txes with feerates as low as 3 sat/vbyte
< sipa> interesting
< az0re> There is more historical mempool/feerate estimate data available here: https://github.com/0e319b6/feerate-estimate-data
< az0re> This is far from the only time I've noticed wildly excessive estimated fees
< az0re> !fees
< gribble> Fee estimates (blocks: fee): (2: 68.414),(4: 8.898), (6: 3.319), (10: 1.138), (20: 1.138), (144: 1.138)
< az0re> OK, no idea how these estimates are calculated, but the super sharp drop off is a red flag
< sipa> may be worth opening an issue for
< darosior> az0re: it's expected. It's possible, but there are not historically enough of them for the estimator to choose this bucket.
< darosior> For example, if you had a new mode -say RECKLESS- with far less probability than ECONOMICAL, then it'd probably give you these estimations.
< az0re> darosior: "not historically enough of them" -- of what?
< phantomcircuit> az0re, the problem is that setting the fee rate too high is a known cost, setting it too low tends to result in people with transactions that are stuck
< phantomcircuit> the latter being much worse for 90% of people
< az0re> phantomcircuit: I totally understand that, and I totally understand being conservative. However:
< az0re> 1. Isn't this why there is CONSERVATIVE and ECONOMICAL? `1 ECONOMICAL` in this case really should not be giving me 200 sat/vbyte!
< az0re> 2. We should still see a smooth drop off. But `1 CONSERVATIVE` and `10 CONSERVATIVE` are identical.
< az0re> And it extends beyond 1 and 10; very frequently I see huge swathes of the estimates sitting at exactly the same feerate
< az0re> If `1 CONSERVATIVE` gave 200, `10 CONSERVATIVE` gave 40, and `1 ECONOMICAL` gave 20 I would have nothing to complain about
< az0re> <phantomcircuit> [...] result in people with transactions that are stuck
< az0re> Also, isn't this why RBF is a thing? :)
< az0re> I understand that doesn't necessarily solve the problem in all cases, but it should reduce the caution for suggesting too-low feerates, especially for ECONOMICAL
< az0re> Finally, I will stop spamming after this I swear, but the bimodal distribution of the mempool seems fundamentally broken to me. In my idealized vision of the world, there would be monotonically increasing mempool pressure with decreasing feerate.
< az0re> The current structure suggests to me that the auction mechanics of the mempool are not really working correctly
< queip> long term, would bitcoind start with 1s/vb and then in background keep slowly RBFing it until it works? In transaction you would specify how much you want to wait, how much you can wait, and absolute limit (so before it it sets very max fee) and the max fee
< queip> e.g. dust consolidations would be like 144,1440,6000, 100[s/vb] while regular onchain shopping could be 3,6,24,1000 for example
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/79a3b59cc706...1dfe19e2840b
< bitcoin-git> bitcoin/master 538be42 Ivan Metlushko: wallet: fix importdescriptor silent fail
< bitcoin-git> bitcoin/master 1dfe19e Wladimir J. van der Laan: Merge #20153: wallet: do not import a descriptor with hardened derivations...
< bitcoin-git> [bitcoin] laanwj merged pull request #20153: wallet: do not import a descriptor with hardened derivations into a watch-only wallet (master...importdesc_silent_fail) https://github.com/bitcoin/bitcoin/pull/20153
< nanotube> sipa: yes, i still run gribble. mempool command fetches data from mempool.space
< nanotube> az0re: fees command pulls from blockstream.info api
< bitcoin-git> [bitcoin] hebasto opened pull request #20357: ci: Use travis_fold on Travis CI only (master...201109-travis) https://github.com/bitcoin/bitcoin/pull/20357
< kanzure> are we using git read-tree or subtree for libsecp256k1?
< sipa> git subtree; i don't know what git read-tree is
< kanzure> is the usage of subtree documented? i'm thinking about extracting test_framework into a python package.
< kanzure> and i would like to preserve commit history if possible
< luke-jr> that's the opposite direction..
< sipa> subtree can preserve history afaik
< sipa> when splitting off a tree
< kanzure> opposite direction should be ok
< sipa> git subtree split may do what you want
< bitcoin-git> [bitcoin] ffontaine opened pull request #20358: src/randomenv.cpp: fix build on uclibc (master...master) https://github.com/bitcoin/bitcoin/pull/20358
< bitcoin-git> [bitcoin] dongcarl opened pull request #20359: depends: Various config.site.in improvements and linting (master...2020-11-config-site-cleanup) https://github.com/bitcoin/bitcoin/pull/20359
< stevenroose> I'm reading "A heap is used so that not all items need sorting if only a few are being sent." in net_processing.cpp.
< stevenroose> I don't understand how the C++ heap implementation works I think, does it order lazily?
< sipa> stevenroose: the STL implementation shouldn't matter
< sipa> it constructs a heap in O(n) time from the input elements
< sipa> then you can extract the top element from it in O(log n) time
< sipa> so if you only need m elements, it's O(n + m*log(n)) work
< sipa> while full sorting would need O(n*log(n)) work
< stevenroose> oooh, I'm reading that's just the way a heap works, didn't know that. didn't know it had those properties, fancy
< sipa> yeah, it actually more of a priority queue algorithm than a sorting algorithm
< sipa> heapsort is first structuring the input into a heap, and then iteratively extracting the top level, shrinking the heap, and using the new space to put the extracted element
< sipa> *top element
< meshcollider> real_or_random: done ^
< * luke-jr> wonders where the generational gap is between "computers take time to do things" and "wtf? why isn't it done yet?"
< stevenroose> sipa: thanks
< luke-jr> (I didn't learn to care about optimisation until maybe ~18yo or so)