< luke-jr>
this seems way too premature; only 2% of the network's upgraded :/
< gmaxwell>
unfortunately, the issue was made public.
< luke-jr>
:/
< gmaxwell>
it wasn't clear if it would propagate or die out, but since it was moderatly easy to discover on your own, even just the rumors of it risked someone exploiting it.
< achow101>
gmaxwell: luke-jr: there's an r/btc thread about someone finding the bug
< gmaxwell>
achow101: it's referenced in the message. 'circulating'
< nanotube>
would it make sense to propose a 'contact' page on bitcoin.org similar to the one on bitcoincore.org? it appears it is non-trivial to find where to privately report security issues unless one knows to go to bitcoincore.org, since earlz had to come asking on #bitcoin-dev and -core-dev for a way to report.
< gmaxwell>
nanotube: :( I feel really uncomfortable with people going to bitcoin.org for that kind of information.
< harding>
nanotube: any page on Bitcoin.org, top menu, Participate, Development, "to report an issue, please see the Bug Reporting page", Responsible Disclosure.
< gmaxwell>
but its there
< gmaxwell>
yea
< nanotube>
yes not ideal, but people probably still go there unless they know to check bitcoincore.org... so, good that it has the bug reporting page in there somewhere.
< echeveria>
there used to be bitcoin-security, but that was handled sort of poorly.
< echeveria>
the contents of it ended up being published when someone stole the old satoshi email address.
< kanzure>
hi. mailing list admin has hit a bug and is unusable at the moment.
< kanzure>
We're sorry, we hit a bug!
< kanzure>
Please inform the webmaster for this site of this problem. Printing of traceback and other system information has been explicitly inhibited, but the webmaster can find this information in the Mailman error logs.
< sipa>
great.
< echeveria>
I was looking at one of the public internet mapping tools for bitcoin core versions. there's a pretty disturbing number of hosts that have 8332 open.
< echeveria>
is there some tool or setup guide that is telling people to open this port? I thought it was pretty difficult (not a single switch) to get Bitcoin Core to bind the RPC interface to 0.0.0.0.
< echeveria>
of 8000 IPv4 nodes, 1142 have a RPC port 8332 that respond to a SYN.
< gmaxwell>
maybe honeypots?
< gmaxwell>
as you note, you must take extra steps to bind..
< echeveria>
I don't think so. they're over a huge number of different hosts, old and new.
< ken2812221_>
MarcoFalke: Your gpg signing key has expired
< Jmabsd>
the code that qualifies and validates a segwit transaction starts on what code locations?
< Jmabsd>
Where is the code that checks the witness merkle root in the coinbase transaction?
< emilengler>
If I download the linux tarball, will I be able to select a download path for the blockchain ?
< luke-jr>
yes, if you know how or use the GUI
< luke-jr>
better topic for #Bitcoin
< emilengler>
Ok I will keep this in mind excuse me
< provoostenator>
I used invalidateblock on a remote node to go back to ~ 475000, but lost the connection after a few hours. The last debug message is from an hour ago, an updatetip down to 508954. It's in a weird state.
< provoostenator>
Memory usage was swinging between 5GB and 8GB. I was able to shut it down via rpc, though the last message was "net thread exit" which sounds like an unclean exit.
< provoostenator>
Restarting the node, now it's "Replaying blocks", "Rolling back ... 542229" and down from there.
< sdaftuar>
provoostenator: what version bitcoind was it?
< provoostenator>
sdaftuar: v0.17.0rc2 (I was actually dumb enough to not upgrade it before doing this)
< provoostenator>
I also don't know if invalidateblock is supposed to work for such a huge rollback. Though if not, then perhaps the documentation should warn against that.
< sdaftuar>
well, i think we do want it to work
< provoostenator>
Also, the RPC call is blocking. Does getting disconnected have any bearing on that?
< sdaftuar>
no, the invalidateblock function should continue even after the rpc client disconnects, i think
< sdaftuar>
i believe if you had waited long enough, it probably would have finished?
< sdaftuar>
but it might be several hours
< provoostenator>
The logs also suggest it continued for about 30 minutes after disconnecting.
< sdaftuar>
disconnecting blocks is heavily disk-bound. when i last looked at it (on different hardware than i use today), i think i noticed i could disconnect on the order of 3-5 blocks/second, on average
< provoostenator>
It got about halfway in 2-3 hours, so indeed it looked like it would have made it.
< provoostenator>
This is an iMac with SSD and plenty of memory.
< sdaftuar>
we used to have an issue where the memory usage could grow sort of unbounded, as disconnected blocks would have their transactions added to the mempool
< sdaftuar>
but that was fixed
< provoostenator>
Yeah the weird thing I noticed is how dbcache kept growing as it was disconnecting.
< sdaftuar>
but your comment about 5-8GB of memory has me slightly concerned
< provoostenator>
The machine has 64 GB so it didn't run out.
< sdaftuar>
what is -dbcache set to?
< provoostenator>
5000, so that's bad
< provoostenator>
The mempool is just the default, so that shouldn't have grown so much, right?
< sdaftuar>
yeah assuming the code works correctly, the mempool's memory usage would have been bounded pretty well
< provoostenator>
Last log entry had cache=4650.7MiB
< sdaftuar>
oh so that seems good then
< provoostenator>
The 5-8 GB RAM usage was an hour after the last log entry, when I reconnected, found through "top".
< sdaftuar>
alright well maybe this is all expected (crappy) behavior. i don't know of any clever ideas to speed up block disconnection, unfortunately.
< provoostenator>
Before I disconnected (a few hours ago) it was doing "UpdateTip: new best ..." in reverse order, as expcted from doing invalidateblock
< provoostenator>
The logs shows it kept doing that for 30 mins after I disconnected from the machine.
< provoostenator>
When I logged back into the machine, bitcoind was still running, using 5-8 GB of RAM (it was actually going up and down in the space of minutes), but log wasn't updating. I then stopped it via rpc and restarted.
< provoostenator>
So it seems it was still doing _something_, despite not logging.
< provoostenator>
"duplicate block" seems to mean that it already processed it, nothing to do with orphans.
< kanzure>
*poke* postmortem email plzkthx
< provoostenator>
Ok, so roll back made it to 509650 and all seems well. Except the node seems to have forgotten I invalidated block 485000, because it jumped right into IBD and is moving forward again.
< provoostenator>
I guess that's because the node doesn't check the full block index at launch.
< provoostenator>
Restarted, now using v0.17.0rc4, doing nanother invalidateblock. Memory usage is almost 2GB higher than cache= shown in the logs, and seems to outpace it.
< provoostenator>
I've also turned off the index.
< gmaxwell>
what are the actual dirty page counts?
< gmaxwell>
we recently realized that OS cached pages in mmaped files show up in res.
< provoostenator>
Also, it's still going even though bitcoin-cli stop said it would stop. I'll look at the dirty page counts...
< provoostenator>
(note to self: do not google "top dirty pages")
< provoostenator>
Also a long list of "mapped file", e.g. many "locks/index/*.ldb"
< gmaxwell>
k, so ~2GB of your resident size is mapped files.
< gmaxwell>
whats your dbcache setting?
< provoostenator>
dbcache=5000 MB, the log currently says cache=2300 MiB, so that part makes sense?
< provoostenator>
It's the just the other 6 GB that needs explaining. Memory usage is now 10 GB. 38 more and the machine is going to OOM, which I'm not going to allow.
< provoostenator>
MALLOC_TINY is now at 8.7, so that seems to be the thing that's mooning.
< provoostenator>
(actually this is still v0.17.0rc2, sorry, though hopefully that doesn't matter here)
< provoostenator>
(no, it is v0.17.0rc4)
< provoostenator>
kill has no effect, kill -9 did
< provoostenator>
Getting fairly consistent behavior now, even with disablewallet=1. bitcoin-cli stop seems to stop the RPC server, but not the invalidation process. Curious if anyone can reproduce. I'll let it sync to the tip before trying again.
< sipa>
if invalidateblock does not succeed, its state isn't writte
< sipa>
it first disconnects the blocks, and then marks them as invalid
< sipa>
so at startup they will be connected again if bitcoind was killed in the middle
< provoostenator>
That makes sense. I wonder if it matter that I was essentially interrupting IBD with that invalidateblock call. Memory usage seemed way worse than what I saw earlier today.
< sipa>
invalidateblock also keeps a list of transactions to re-add to the mempool after the invalidation completes
< sipa>
i assume that's the memory usage you see
< provoostenator>
Is it also not abortable once in progress?
< sipa>
no
< provoostenator>
Ok, so in that case the way to roll back a long way would be to do it in smaller increments.
< sipa>
i was expecting the code to be elsewhere, my bad
< provoostenator>
I take great pride in doing stupid things that lead to a new release candidate, so hopefully you'll find something :-)
< gmaxwell>
provoostenator: are you running txindex?
< provoostenator>
No, I did the first time today, but turned that off in more recent attempts.
< provoostenator>
Re incremental approach: I rolled back ~10,000 blocks using about 13GB of RAM, cache=2200 at the peak. Sounds like it's holding all transactions in memory.
< provoostenator>
But then it gets weird. ERROR: AcceptToMemoryPoolWorker: Consensus::CheckTxInputs: ... bad-txns-premature-spend-of-coinbase, tried to spend coinbase at depth 92
< provoostenator>
InvalidChainFound: invalid block [the block I invalidated]
< gmaxwell>
thats normal.
< provoostenator>
Yeah, but then it starts syncing again.
< provoostenator>
Ok, now I think I destroyed my chain :-) At boot: "assertion failed: (!setBlockIndexCandidates.empty()), function PruneBlockIndexCandidates, file validation.cpp, line 2547"
< sipa>
provoostenator: i found the issue
< sipa>
it's specific to InvalidateBlock
< provoostenator>
sipa: nice!
< provoostenator>
sipa: is it because disconnectpool holds on to transactions which reference a shared_ptr<CBlock> pblock, so those don't get deallocated?
< sipa>
provoostenator: no
< sipa>
the event queue holds on to the shared_ptr<CBlock> objects in callbacks to DisconnectedBlock
< sipa>
and InvalidateBlock doesn't limit the size of the queue
< sipa>
provoostenator: could you check whether this issue also occurs when Rewinding?
< sipa>
create a 0.13.0 node, sync it to tip, and then upgrade to 0.17+
< sipa>
i suspect it is, and if that's the case, i would consider it a release blocker
< provoostenator>
That's the rewind that happens if you upgrade a non-segwit node to a segwit node?
< sipa>
yup
< provoostenator>
I'll give it a try this weekend or early next week. Getting a bit late here. Maybe someone else gets to it first.
< sipa>
thanks!
< provoostenator>
I'm not looking forward to doing another release notes for 0.14 and 0.15 backports :-)
< MarcoFalke>
About #14289, was it ever supported to call invalidateblock on a block very far back?