< da2ce7> Happy new year! :)
< btcdrak> wumpus: can we tag 0.13.2? :-)
< wumpus> btcdrak: yes
< bitcoin-git> [bitcoin] laanwj pushed 1 new commit to master: https://github.com/bitcoin/bitcoin/commit/1d2d67692c74a5539f3736754d84f0aa6a702c56
< bitcoin-git> bitcoin/master 1d2d676 Wladimir J. van der Laan: qt: Set transifex slug to 0.14...
< bitcoin-git> [bitcoin] jonasschnelli pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/1d2d67692c74...53442af0aac3
< bitcoin-git> bitcoin/master 09aefb5 Cory Fields: build: Fix 'make deploy' for OSX...
< bitcoin-git> bitcoin/master b01667c Jonas Schnelli: Mention RSVG dependency when creating the disk image on OSX
< bitcoin-git> bitcoin/master 2fb98f6 Don Patterson: Fix bug in dmg builder so that it actually reads in the configuration file
< bitcoin-git> [bitcoin] jonasschnelli closed pull request #9412: build: Fix 'make deploy' for OSX (master...2016/12/fix_mac_deploy) https://github.com/bitcoin/bitcoin/pull/9412
< bitcoin-git> [bitcoin] laanwj pushed 1 new commit to 0.13: https://github.com/bitcoin/bitcoin/commit/0d719145b018e28d48d35c2646a5962b87c60436
< bitcoin-git> bitcoin/0.13 0d71914 Wladimir J. van der Laan: doc: Remove ... from release notes
< wumpus> * [new tag] v0.13.2 -> v0.13.2
< jonasschnelli> \o/
< wumpus> *almost* new year tag :p
< jouke_> I updated a .10 node to .13.1. It connects to an other .13.1 node, but it doesn't seem to sync the blockchain. But when I look at the blockheight, it might already got "stuck" when it was a .10 node. I updated it, because some rpc calls were a bit slow.
< jouke_> In debug log I see that it is receiving "invs".
< gmaxwell> jouke_: if you were stuck it is presumably because corruption made you reject a block. you can try running 'reconsiderblock <block hash after your current height>'
< jouke_> I retreived the blockhash from an other node, but that doesn't seem to do anything.
< gmaxwell> jouke_: I assume you don't have historic logs from the point where it got stuck?
< jouke_> no, I don't unfortunately
< jouke_> It just received information about the latest block and I see it is retreiving and receiving the headers of that block, but that's it.
< jouke_> Hmm, we use that node to gather information about mempool size, and it stopped accepting blocks at around the 29th, but it did receive transactions which it added to it's mempool.
< gmaxwell> jouke_: do you have logs that cover the 29th?
< jouke_> no
< gmaxwell> the reconsiderblock did nothing? can you try invalidateblock <thatblock> then reconsider?
< gmaxwell> can you also run getchaintips and pastebin the results?
< jouke_> And thatblock = currentheight+1 right?
< gmaxwell> right what is your current height?
< jouke_> It's stuck at 445623
< gmaxwell> can you getblockhash for 445623 ?
< jouke_> 000000000000000002c45ed9057608b268a447499fcd8bea8e7930bfed11eaaf
< gmaxwell> jouke_: okay run reconsiderblock 000000000000000000fc97c4cd1f956f46a420536ca4851c31ad9d32f4e8ec0c
< jouke_> already did that
< jouke_> Didn't do anything
< gmaxwell> jouke_: can you try doing a getblock 000000000000000000fc97c4cd1f956f46a420536ca4851c31ad9d32f4e8ec0c and see if it says anything? grep your logs for that hash and see if there is any mention?
< gmaxwell> unfortunately I think there are types of rejection that reconsider will not reconsider, it's really just intended to be the compliment to invalidateblock
< jouke_> error code: -32603
< jouke_> error message:
< jouke_> Can't read block from disk
< jouke_> chaintips: https://www.zerobin.net/?70af58e5db340709#1i/w7FQhQE0ubu1+9QO5ANJPRSJ+V24C0LgiYVl15gU=
< gribble> https://github.com/bitcoin/bitcoin/issues/1 | JSON-RPC support for mobile devices ("ultra-lightweight" clients) · Issue #1 · bitcoin/bitcoin · GitHub
< gmaxwell> jouke_: okay your blockchain data is corrupted. the block isn't on your disk (or is corrupted on disk) but is in the blockindex.
< jouke_> Hmm. nothing strange in syslog about file corruption or anything.
< gmaxwell> any chance you had a unclean power loss? the block isn't added into the block index until its written to disk and fsynced, but if your system is ignoring fsyncs or there is some other misbehavior in your storage it could be lost during a power failure.
< jouke_> no, system has been up for a whie
< jouke_> while
< gmaxwell> oh this was 0.10 before?
< jouke_> yes
< gmaxwell> ah ha.
< gmaxwell> I believe we actually had a bug that could cause that before.
< jouke_> I was already looking through the issues, but I did not come across it yet.
< gmaxwell> jouke_: in any case, I believe there is no other resolution right now to your condition but to reindex. setting a higher dbcache than default, if you have the ram will make it run much faster.
< jouke_> Shame I don't have the logs. I am making a note to push all bitcoin node logs to a log server.
< jouke_> gmaxwell: thanks for the help!
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #9457: WIP: [qt] translate bitcoin.cpp and libbitcoin_common_a_SOURCES (master...Mf1701-qtTrans) https://github.com/bitcoin/bitcoin/pull/9457
< jonasschnelli> Hmm... full block SPV with random peers is horrible slow. 10-12 blocks per minute around height 445100
< jonasschnelli> (on a 100MBit link)
< jonasschnelli> Maybe I just had bad luck with remote peers
< gmaxwell> increase your socket recieve buffer maximum size.
< gmaxwell> by like a factor of 10, you'll likely see it much faster.
< gmaxwell> Does your fetching code fetch in parallel?
< jonasschnelli> Yes. Same code.
< jonasschnelli> Maybe I need to unset NODE_NETWORK?
< jonasschnelli> debug=net shows plenty of tx invs
< gmaxwell> increase your socket recieve buffer maximum size.
< gmaxwell> One problem (no doubt there is more) is that it cannot achieve high throughput for transfers because the recieve buffers are so small that almost any processing delay and they fill.
< jonasschnelli> gmaxwell: Thanks. I'll try that.
< jonasschnelli> gmaxwell: -maxreceivebuffer=50000 = 1block per second. Better,... need to profile.. but I guess SyncTransaction with IsMine, etc. is the bottleneck
< gmaxwell> thanks for checking and, it sounds like, validating my theory. Not just how long sync takes, but the block deseralization isn't fast, ans I assume your sync runs in order, which means that when blocks arrive out of order and eventually the missing one shows up, it is probably processing many blocks at once, which creates a long delay in the message handling loop which will stall the recieve buf
< gmaxwell> fers.
< jonasschnelli> The blockdownload im using (#9171) does download in parallel, then store them on the disk and, process them in order (unserialize again probably).
< gribble> https://github.com/bitcoin/bitcoin/issues/9171 | Introduce auxiliary block requests by jonasschnelli · Pull Request #9171 · bitcoin/bitcoin · GitHub
< gmaxwell> right, and say that you recieve blocks 0 2 3 4 5 6 7 8 9 1 will it end up processing 9 of them at the moment it recieves block 1 in the message handler? if so... it is not emptying the recieve buffers during that time.
< jonasschnelli> gmaxwell: I think it does empty the receive buffer,... ProcessNewBlock gets called for all blocks where only the header gets verified, the stored to the disk. I guess this drains the buffer...
< jonasschnelli> *then
< gmaxwell> the recieve buffer is drained as the blocks are saved to disk. it saves 1 to disk but then processes it and then will connect 1 2 3 4 5 6 7 8 9 all within that function call.
< gmaxwell> and during that time the message handler is not handling any messages. :)
< gmaxwell> so the recieve buffers fill.
< jonasschnelli> hmmm...
< jonasschnelli> Ah. Right... all the processing (LoadBlock, SyncTransaction, etc.) is done in thread "bitcoin-msghandler".
< gmaxwell> which fine, but not if it's processing a bunch of blocks at once.
< gmaxwell> e.g. it should just process two and then return control to the message handler loop.
< jonasschnelli> Okay.. I'll try that. Is 16 atm.
< jonasschnelli> *It's
< gmaxwell> where is it limited to 16? like.. if you get 0 2 3 4 5 ... 999 1 doesn't end up processing 1000 at once?
< jonasschnelli> I guess an explicit thread "process spv" (or similar) doesn't make that much sense. SyncTransaction does hold cs_main anyways.
< jonasschnelli> 16 is the limit for the max amount of blocks per auxiliary request
< jonasschnelli> If block 1 comes in last, it processes 15 blocks in a row.
< jonasschnelli> Once the 16 are processed, it continues with the next batch of 16
< jonasschnelli> (through findNextBlocksToDownload())
< jonasschnelli> Ah. no, MAX_BLOCK_TO_PROCESS_PER_ITERATION == 5
< gmaxwell> oh that isn't many blocks to have in flight at once.
< gmaxwell> it should be possible to have many in flight but only process a few at a time.
< gmaxwell> if you cut the number in flight your throughput will decrease due to round trip delays.
< jonasschnelli> I just got in 16 blocks nice in order. It took more then 5s per block. I think its a slow peer.
< gmaxwell> thats one of the reasons you want to have lots of blocks in flight at once. :)
< jonasschnelli> Okay. Let me try that.
< jonasschnelli> 16 is max per peer. Maybe the request should contain 16*16 or similar.
< gmaxwell> yes 16 max per peer is probably fine.
< jonasschnelli> gmaxwell: Oh yes. That was the bottleneck, not enough blocks in flight, parallelism was not ideal
< jonasschnelli> now 450 blocks in 1min.
< jonasschnelli> (at height 445761)
< MarcoFalke> wumpus: No, the translations were never added back
< MarcoFalke> As of today we are missing the ones that were moved to warnings.cpp
< MarcoFalke> (additionally)
< MarcoFalke> I don't understand the translation process code enough to fix this, but something like #9457 should do it
< gribble> https://github.com/bitcoin/bitcoin/issues/9457 | WIP: [qt] translate bitcoin.cpp and libbitcoin_common_a_SOURCES by MarcoFalke · Pull Request #9457 · bitcoin/bitcoin · GitHub
< gmaxwell> oh interesting, I didn't really realize translations were file specific, I thought they were string to string conversions?
< MarcoFalke> It is an issue with some files are not picked up by the translation process code
< gmaxwell> oh I see.
< * BlueMatt> is willing to review in exchange for review on #9289, #9419 or #9375 :)
< gribble> https://github.com/bitcoin/bitcoin/issues/9289 | net: drop boost::thread_group by theuni · Pull Request #9289 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/9419 | Stop Using cs_main for CNodeState/State() by TheBlueMatt · Pull Request #9419 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/9375 | Relay compact block messages prior to full block connection by TheBlueMatt · Pull Request #9375 · bitcoin/bitcoin · GitHub
< gmaxwell> Is there a reason that make check doesn't run qa/pull-tester/rpc-tests.py ?
< gmaxwell> the tests in that are far more meaningful than most of the unit tests, IMO.
< BlueMatt> someone was complaining the other day that there is no way (afaik) to run rpc-tests from make ...
< f0nky> hi all, can i ask u guys a question?
< achow101> are the 0.13.2 detached sigs ready yet?
< cfields> achow101: running sanity tests. eta ~10 min.
< cfields> gitian builders: 0.13.2 detached sigs pushed
< luke-jr> crap, forgot to save a copy of the intermediate files
< bitcoin-git> [bitcoin] isle2983 opened pull request #9459: Improvements to copyright_header.py and some minor copyright header tweaks. (master...PR-copyright-script-improve) https://github.com/bitcoin/bitcoin/pull/9459
< cfields> BlueMatt: right, so does making CNodeState::cs static not enforce the only-one-nodestate-at-once rule?