< bitcoin-git>
bitcoin/0.13 0d71914 Wladimir J. van der Laan: doc: Remove ... from release notes
< wumpus>
* [new tag] v0.13.2 -> v0.13.2
< jonasschnelli>
\o/
< wumpus>
*almost* new year tag :p
< jouke_>
I updated a .10 node to .13.1. It connects to an other .13.1 node, but it doesn't seem to sync the blockchain. But when I look at the blockheight, it might already got "stuck" when it was a .10 node. I updated it, because some rpc calls were a bit slow.
< jouke_>
In debug log I see that it is receiving "invs".
< gmaxwell>
jouke_: if you were stuck it is presumably because corruption made you reject a block. you can try running 'reconsiderblock <block hash after your current height>'
< jouke_>
I retreived the blockhash from an other node, but that doesn't seem to do anything.
< gmaxwell>
jouke_: I assume you don't have historic logs from the point where it got stuck?
< jouke_>
no, I don't unfortunately
< jouke_>
It just received information about the latest block and I see it is retreiving and receiving the headers of that block, but that's it.
< jouke_>
Hmm, we use that node to gather information about mempool size, and it stopped accepting blocks at around the 29th, but it did receive transactions which it added to it's mempool.
< gmaxwell>
jouke_: do you have logs that cover the 29th?
< jouke_>
no
< gmaxwell>
the reconsiderblock did nothing? can you try invalidateblock <thatblock> then reconsider?
< gmaxwell>
can you also run getchaintips and pastebin the results?
< gmaxwell>
jouke_: okay run reconsiderblock 000000000000000000fc97c4cd1f956f46a420536ca4851c31ad9d32f4e8ec0c
< jouke_>
already did that
< jouke_>
Didn't do anything
< gmaxwell>
jouke_: can you try doing a getblock 000000000000000000fc97c4cd1f956f46a420536ca4851c31ad9d32f4e8ec0c and see if it says anything? grep your logs for that hash and see if there is any mention?
< gmaxwell>
unfortunately I think there are types of rejection that reconsider will not reconsider, it's really just intended to be the compliment to invalidateblock
< gmaxwell>
jouke_: okay your blockchain data is corrupted. the block isn't on your disk (or is corrupted on disk) but is in the blockindex.
< jouke_>
Hmm. nothing strange in syslog about file corruption or anything.
< gmaxwell>
any chance you had a unclean power loss? the block isn't added into the block index until its written to disk and fsynced, but if your system is ignoring fsyncs or there is some other misbehavior in your storage it could be lost during a power failure.
< jouke_>
no, system has been up for a whie
< jouke_>
while
< gmaxwell>
oh this was 0.10 before?
< jouke_>
yes
< gmaxwell>
ah ha.
< gmaxwell>
I believe we actually had a bug that could cause that before.
< jouke_>
I was already looking through the issues, but I did not come across it yet.
< gmaxwell>
jouke_: in any case, I believe there is no other resolution right now to your condition but to reindex. setting a higher dbcache than default, if you have the ram will make it run much faster.
< jouke_>
Shame I don't have the logs. I am making a note to push all bitcoin node logs to a log server.
< jouke_>
gmaxwell: thanks for the help!
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #9457: WIP: [qt] translate bitcoin.cpp and libbitcoin_common_a_SOURCES (master...Mf1701-qtTrans) https://github.com/bitcoin/bitcoin/pull/9457
< jonasschnelli>
Hmm... full block SPV with random peers is horrible slow. 10-12 blocks per minute around height 445100
< jonasschnelli>
(on a 100MBit link)
< jonasschnelli>
Maybe I just had bad luck with remote peers
< gmaxwell>
increase your socket recieve buffer maximum size.
< gmaxwell>
by like a factor of 10, you'll likely see it much faster.
< gmaxwell>
Does your fetching code fetch in parallel?
< jonasschnelli>
Yes. Same code.
< jonasschnelli>
Maybe I need to unset NODE_NETWORK?
< jonasschnelli>
debug=net shows plenty of tx invs
< gmaxwell>
increase your socket recieve buffer maximum size.
< gmaxwell>
One problem (no doubt there is more) is that it cannot achieve high throughput for transfers because the recieve buffers are so small that almost any processing delay and they fill.
< jonasschnelli>
gmaxwell: Thanks. I'll try that.
< jonasschnelli>
gmaxwell: -maxreceivebuffer=50000 = 1block per second. Better,... need to profile.. but I guess SyncTransaction with IsMine, etc. is the bottleneck
< gmaxwell>
thanks for checking and, it sounds like, validating my theory. Not just how long sync takes, but the block deseralization isn't fast, ans I assume your sync runs in order, which means that when blocks arrive out of order and eventually the missing one shows up, it is probably processing many blocks at once, which creates a long delay in the message handling loop which will stall the recieve buf
< gmaxwell>
fers.
< jonasschnelli>
The blockdownload im using (#9171) does download in parallel, then store them on the disk and, process them in order (unserialize again probably).
< gmaxwell>
right, and say that you recieve blocks 0 2 3 4 5 6 7 8 9 1 will it end up processing 9 of them at the moment it recieves block 1 in the message handler? if so... it is not emptying the recieve buffers during that time.
< jonasschnelli>
gmaxwell: I think it does empty the receive buffer,... ProcessNewBlock gets called for all blocks where only the header gets verified, the stored to the disk. I guess this drains the buffer...
< jonasschnelli>
*then
< gmaxwell>
the recieve buffer is drained as the blocks are saved to disk. it saves 1 to disk but then processes it and then will connect 1 2 3 4 5 6 7 8 9 all within that function call.
< gmaxwell>
and during that time the message handler is not handling any messages. :)
< luke-jr>
crap, forgot to save a copy of the intermediate files
< bitcoin-git>
[bitcoin] isle2983 opened pull request #9459: Improvements to copyright_header.py and some minor copyright header tweaks. (master...PR-copyright-script-improve) https://github.com/bitcoin/bitcoin/pull/9459
< cfields>
BlueMatt: right, so does making CNodeState::cs static not enforce the only-one-nodestate-at-once rule?