< GitHub190> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/fc4f4547b7f6...14b7b3fb9f79
< GitHub190> bitcoin/master fa4bfb4 MarcoFalke: [wallet, policy] ParameterInteraction: Don't allow 0 fee
< GitHub190> bitcoin/master 14b7b3f Wladimir J. van der Laan: Merge #8814: [wallet, policy] ParameterInteraction: Don't allow 0 fee...
< GitHub50> [bitcoin] laanwj closed pull request #8814: [wallet, policy] ParameterInteraction: Don't allow 0 fee (master...Mf1607-walletHighFeeWarn) https://github.com/bitcoin/bitcoin/pull/8814
< jl2012> when I'm working on rpctests, sometimes the node will generate empty blocks despite the mempool is not empty. It usually happens when the transactions in mempool are big. Why would that happen and how could I force it to mine those txs?
< gmaxwell> will it continue to mine empty blocks, not just a one time event?
< sipa> jl2012: perhaps an interaction between max block size an cpfp mining?
< sipa> if you build a chain of dependent transactions that does not actually fit in a block
< gmaxwell> and then there is nothing else to include other than that chain.
< jl2012> all parents are confirmed. And keep mining empty blocks
< jl2012> oh, not really keeping mining empty blocks forever
< jl2012> there is only 1 tx in mempool, but it'd cleanup the mempool only if I generate about 500 blocks
< luke-jr> fee too low, so it's waiting for priority?
< sipa> anything particular about this tx?
< jl2012> about 80kB
< jl2012> luke-jr: so it'd rather mining empty blocks, than mining a tx with insufficient fee?
< luke-jr> jl2012: it shouldn't happen with the current code afaik, but that's what it sounds like if it's mining it after blocks
< luke-jr> I assume there's no locktime?
< jl2012> no locktime
< sipa> so after 500 blocks, it does mine it?
< jl2012> yes, about 400 blocks. I'm try to find the exact number
< gmaxwell> "Interesting"
< sipa> is the height it activates a nice multiple of 144 after the activation of segwit?
< jl2012> well, I find something more strange
< gmaxwell> whats that?
< jl2012> I have a loop. Each round it generates 1 block, and print the results of getrawmempool
< luke-jr> so it's being pruned and then re-accepted? O.o
< jl2012> i will push the code, wait
< sipa> jl2012: heh
< jl2012> it was it's just generate 1 block, print the mempool
< jl2012> nothing strange
< jl2012> from the output i assume it was mined at the 11th block, but it re-appeared after that
< sipa> that's not possible...
< sipa> or shouldn't be possible
< sipa> its inputs are spent after mining it
< jl2012> by the way the tx is not segwit
< jl2012> maybe i should really see if it is really mined. But that's kind of random
< sipa> you can call gettransaction to see its number of confirmations, if the transaction applies to the local wallet
< jl2012> that's not. But I could read the content of blocks
< gmaxwell> how do these tests mine?
< gmaxwell> IIRC getblocktemplate can cache templates.
< jl2012> gmaxwell, the mempool has only 1 tx, if the mempool is cleaned up after mining one block, the tx is not mined (i suppose)
< warren> btcdrak: wow
< btcdrak> warren: I assume you are referring to this "<cfields_> wumpus: i've added thin toolchain builders to depends that rid us of the ubuntu toolchain dependency. Only needed once for bootstrap, then we're self-hosted"
< warren> cfields_: can the same binary result be reached from another distro's toolchain?
< phantomcircuit> luke-jr: there seems to be a bug in getblocktemplate
< phantomcircuit> running master
< phantomcircuit> "coinbasevalue": 333038622,
< phantomcircuit> 2016-09-28 10:43:08 ERROR: ConnectBlock(): coinbase pays too much (actual=332777360 vs limit=312500000)
< phantomcircuit> (this is on testnet)
< sipa> phantomcircuit: ugh!
< phantomcircuit> or possibly im just not including transactions?
< luke-jr> phantomcircuit: in or outside the context of jl2012's stuff?
< phantomcircuit> hmm
< phantomcircuit> this seems like im not including transactions actually
< sipa> phantomcircuit: that would explain
< phantomcircuit> im just guessing based on the difference
< jl2012> sorry, seems just a bug in my code
< luke-jr> >_<
< jl2012> false alarm
< sipa> jl2012: goox
< sipa> good
< jl2012> I have 2 nodes. I generated blocks in one, and the other one is not synced
< jl2012> so sometimes it will generate a block with the new tx, but the block is then get orphaned
< jl2012> and the tx is returned to the mempool
< gmaxwell> jl2012: oh obvious. Good.
< wumpus> should I move/clone https://github.com/laanwj/bitcoin-maintainer-tools into bitcoin-core?
< sipa> i wonder if we shouldn't try to integrate a buildinfo-less mode into the source tree itself
< sipa> ./configure --disable-buildinfo
< wumpus> maybe, but this is just an experiment for now
< wumpus> I don't think that's an argument against the tool though?
< sipa> sure, just making an unrelated comment
< sipa> but in general of things are useful for the project jn general, i think they should be in repo
< wumpus> having it outside the source tree has its advantages, it means easier experimentation, quicker iteration etc
< wumpus> not everything having to go through series of ACK
< wumpus> these are just my private tools anyhow
< wumpus> I think I tend to disagree with that in practice, not everything has to be in the repo
< wumpus> sure, if something is needed all the time like the github-merge script it makes sense to have it in the repo
< wumpus> although even of that I have my own version, which I incidentally make improvements to, once there's enough I'll file a PR. But having to do that all the time gives a lot of overhead for everyone involved.
< wumpus> I have some other scripts too, a few other code analysis tools, and to build release notes, author lists etc. All can be useful, but it's pretty much a meta-project to bitcoin itself.
< wumpus> it isn't part of the bitcoin core release cycle either
< wumpus> changes can happen any time, even between rcs, or between rc and final
< sipa> fair enough
< sipa> no objection to moving it to bitcoin-core bte
< sipa> *btw
< wumpus> thanks
< wumpus> I also think we can just be more liberal with changes in a tools repo. If you think something is useful, just do it, no need for long review cycles etc.
< sipa> right, agree
< wumpus> (or maintaining a listof 130 PRs *despairs*)
< jonasschnelli> BlueMatt: A nice... was looking through the code and couldn't find the Auth/MAC
< BlueMatt> jonasschnelli: in UDPMessageHeader - chk1 and chk2
< BlueMatt> those are just the hash of the message = the magic (yea, yea, not how you're supposed to do a MAC....whatever)
< BlueMatt> s/=/+/
< * jonasschnelli> is checking the code
< BlueMatt> see FillChecksum
< jonasschnelli> need to checkout first,... github not searchable on forks. :(
< BlueMatt> its all in one file, you can just read that :p
< jonasschnelli> wasn't aware of udpnet.cpp :)
< jonasschnelli> FillChecksum(std::get<3>(msg), std::get<1>(msg), std::get<2>(msg));
< jonasschnelli> std::get<3>(msg) is a key or something?
< sipa> it gets the 3rd element of a tuple
< jonasschnelli> Yes.
< jonasschnelli> its declared as a uint64_t
< jonasschnelli> Wonder how it gets popuplated
< jonasschnelli> Or is FillChecksum(uint64_t magic, UDPMessage& msg, const unsigned int length) similar to our TCP p2p message packeges where magic is the network magic?
< jonasschnelli> Okay. Got it...
< jonasschnelli> UDPConnectionInfo.local_magic gets populated through addudpnode
< jonasschnelli> Yes. Thats a nice simple auth
< wumpus> yes
< jonasschnelli> Maybe I missed something: whats the reason why importmulti requires a scriptPubKey https://github.com/bitcoin/bitcoin/pull/7551/files#diff-522490d83dce5375d423b23886e4125eR979?
< jonasschnelli> Can't I just import pubkeys?
< GitHub61> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/14b7b3fb9f79...7b05af634be4
< GitHub61> bitcoin/master ac01ff2 Wladimir J. van der Laan: doc: Add privacy recommendation when running hidden service
< GitHub61> bitcoin/master 7b05af6 Wladimir J. van der Laan: Merge #8823: doc: Add privacy recommendation when running hidden service...
< GitHub121> [bitcoin] laanwj closed pull request #8823: doc: Add privacy recommendation when running hidden service (master...2016_09_tor_recommendation) https://github.com/bitcoin/bitcoin/pull/8823
< sipa> jonasschnelli: yes, you can, but it's hard to create an API that always has expected results
< sipa> we have so many different ways through which we can start treating a particular script as ours
< sipa> maybe it's an imported address
< sipa> maybe it is p2sh for a redeemscript that contains something we know
< sipa> maybe we have the private key
< sipa> maybe we have a pubkey
< sipa> i think it's impossible to explain all those conditions
< sipa> so what does 'importing a pubkey' mean
< jonasschnelli> Yes. I see your point.
< sipa> should it make sends to that pubkey spendable? or also p2pkh involving those keys? what about a 1-of-3 multisig that involves those keys?
< sipa> by forcing the user to state exactly what they want, you avoid that
< sipa> and you can check whether you can provide the functionality they're asking for
< jonasschnelli> But explicit requiring the scriptPubKey could be a little bit annoying if you just want to import a P2PKH key?!
< sipa> it's a compromise between convenience and clarity, yes
< jonasschnelli> A wait... there is an "address" option for the scriptPubKey... okay. let me continue the testing..
< jonasschnelli> Makes sense so far.
< sipa> also, if you just want a p2pkh, you should import just that address, and set watchonly
< jonasschnelli> Agree. A plain pubkey import is kind of unspecified.
< sipa> the idea is that 1) you state what you want to see made spendable/solvable/watching, and then 2) all necessary information to accomplish that (which may mean giving redeemscripts, or pubkeys, or private keys)
< sipa> and in the simplest cases, (2) is empty
< GitHub190> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7b05af634be4...dc641415e75e
< GitHub190> bitcoin/master 5776e8a fanquake: [depends] Fix Qt compilation with Xcode 8
< GitHub190> bitcoin/master dc64141 Wladimir J. van der Laan: Merge #8820: [depends] Fix Qt compilation with Xcode 8...
< GitHub22> [bitcoin] laanwj closed pull request #8820: [depends] Fix Qt compilation with Xcode 8 (master...depends-qt-xcoderun) https://github.com/bitcoin/bitcoin/pull/8820
< GitHub163> [bitcoin] paveljanik opened pull request #8826: Do not include env_win.cc on non-Windows systems (master...20160928_leveldb_no_win) https://github.com/bitcoin/bitcoin/pull/8826
< jonasschnelli> wumpus: IMO this is ready: https://github.com/bitcoin/bitcoin/pull/7551
< jonasschnelli> Maybe the help text is not perfect...
< wumpus> awesome
< wumpus> well, that can be improved later
< sipa> i'd like to review 7551 as well, but haven't found the time yet
< wumpus> ok
< jonasschnelli> Yes. No hurry.
< jonasschnelli> We should just make sure we merge is some month before 0.14 RC
< sipa> absolutely
< wumpus> yes, it is functionality that certainly needs to be in master for a while before the release
< wumpus> it's complex enough, I'm sure there are problems that will only be found in actual use
< jonasschnelli> Yes. Its relatively complex and could lead to coins lost in the worst case.
< wumpus> we reallly need it though. After a while we can then start to deprecate the whole zoo of other import calls.
< GitHub167> [bitcoin] sdaftuar opened pull request #8827: [qa] Split up slow RPC calls to avoid pruning test timeouts (master...fix-pruning-timeout) https://github.com/bitcoin/bitcoin/pull/8827
< GitHub163> [bitcoin] pstratem opened pull request #8828: Move CWalletDB::ReorderTransactions to CWallet (master...2016-09-28-cwallet-reordertransactions) https://github.com/bitcoin/bitcoin/pull/8828
< GitHub184> [bitcoin] jnewbery opened pull request #8829: Add bitcoin-tx JSON tests (master...test-bitcoin-tx-json) https://github.com/bitcoin/bitcoin/pull/8829
< GitHub21> [bitcoin] jnewbery opened pull request #8830: [test] Add option to run bitcoin-util-test.py manually (master...test-bitcoin-util-manually) https://github.com/bitcoin/bitcoin/pull/8830
< pedrobranco> Hi, anyone is experiencing high disk space usage when running bitcoin tests (env. is osx)?
< wumpus> what is 'high disk space usage' these days?
< pedrobranco> 62gb in temporary test folder
< wumpus> for the RPC tests I guess, not the unit tests?
< wumpus> does it at least clean up everything again? the RPC tests have recently been changed to execute tests in parallel, this also means that more disk space is required for the instances running at the same time
< pedrobranco> rpc tests only
< wumpus> you can reduce the parallelism with some command line option if it's too much
< pedrobranco> i have 80G free space and it uses it all before ending rpc tests.
< gmaxwell> 62GB sounds like a lot, do the rpc tests really do that? seems implausable to me.
< wumpus> try rpc-tests.py -parallel=1
< pedrobranco> sure
< GitHub87> [bitcoin] pstratem opened pull request #8831: Replace CWalletDB::ReadKeyValue with CWallet::LoadKeyValue (master...2016-09-28-cwallet-loadkeyvalue) https://github.com/bitcoin/bitcoin/pull/8831
< wumpus> I also doubt it uses tht much in /tmp though, I only have 22GB free there and never had issues
< wumpus> not sure how much the travis VMs have but it should be a comparable amount, not a huge lot
< phantomcircuit> can someone check the travis issue on #8828 ?
< phantomcircuit> tests pass locally
< wumpus> or are these the 'extended tests'?
< phantomcircuit> oh wait they dont
< phantomcircuit> huh
< phantomcircuit> they *did*
< pedrobranco> with rpc-tests.py -parallel=1 i'm having the same issue. p2p-fullblocktest.py is creating the 65G folder
< pedrobranco> and fails on the test #133
< wumpus> which file is so big?
< wumpus> in your pastebin there's 65G test3jvyn0m1/45 and everything below it is *much* smaller
< wumpus> so it must be a file in the top levele dir
< pedrobranco> to continue the tests i had to removed the folder, but i will repeat to show a full disk usage.
< pedrobranco> *remove
< pedrobranco> still running tests...
< arubi> pedrobranco, out of curiosity, can you tell which test? run `top`? should be part of the command line
< wumpus> p2p-fullblocktest
< pedrobranco> ^
< arubi> oh, I thought it was /still/ running, sorry
< pedrobranco> np
< wumpus> I had the same error on openbsd, but that was due to linking a crappy ndb implementation that only supports small files
< wumpus> test is still running here, disk usage looks fairly stable around ~2GB, at subtest 94 now
< wumpus> so I still wonder what file is growing so big there - I guess blocks.db?
< wumpus> (as it's above the node directories)
< pedrobranco> i will now run p2p-fullblocktest only
< wumpus> final size is 2.1G /tmp/testyxoclyo1/14241 for p2p-fullblocktest.py -nocleanup
< wumpus> blocks.db ends up at 1.1G, the rest goes to the node0 dir. So it must be something local there that causes some file to balloon, do you have any changes compared to master?
< jonasschnelli> I hope libevent does a 2.1.x release before our 0.14 (https://github.com/libevent/libevent/releases/)
< wumpus> jonasschnelli: they have done 2.1.x releases, they're just in the beta phase
< jonasschnelli> Yes. I meant a stable one... so package managers like brew (on OSX) can pick this up
< wumpus> (finally, since last year or so, 2.1 has been in dev/alpha since 2009 or so...)
< pedrobranco> it is /var/folders/kt/t6rd1zxx7px8273wlb56c4340000gn/T/testo0lm0oh1/0/blocks.db that has 65G
< wumpus> pedrobranco: interesting. What OS, what version of python do you have?
< wumpus> what does 'import dbm.ndbm; print(dbm.ndbm.library)' in python3 print?
< pedrobranco> OSX 10.11.6 with xcode 7.3.1 , python -V = 2.7.12, python3 -V = 3.5.2
< pedrobranco> let me see
< pedrobranco> prints 'GNU gdbm'
< wumpus> uh oh
< wumpus> if it prints that on a BSD it probably means it is linked against the system's ndbm, BSDs have a ndbm implementation in their libc
< wumpus> in openbsd that's a very silly one that can handle up to 64kb files or so :) but on OSX probably different
< wumpus> let me find something to try for you
< pedrobranco> thanks for the help wumpus, but i need to leave now. i'll be back in a couple of hours.
< wumpus> pedrobranco: see https://github.com/bitcoin/bitcoin/issues/8605 , that's my openbsd problem, if you replace dbm.ndbm with dbm.dumb it will probably work fine
< wumpus> (in qa/rpc-tests/test-framework/blockstore.py)
< wumpus> jonasschnelli: you're on OSX right? did you ever notice p2p-fullblocktest using exorbitant disk space?
< jonasschnelli> I can't remember when i last ran this test...
< jonasschnelli> let me try
< jonasschnelli> (currently compiling, will start in a min)
< wumpus> well it's part of the default pulltester run
< jonasschnelli> Yes. Never noticed big data dirs...
< jonasschnelli> But I have a 1GB SSD..
< jonasschnelli> TB
< jonasschnelli> :-)
< * jonasschnelli> started ./qa/rpc-tests/p2p-fullblocktest.py --nocleanup
< wumpus> BTW bitcoin core on FreeBSD nearly works out of the box with ports-provided development packages, not nearly the struggle it is on OpenBSD. Though I don't think I've run the RPC tests yet :)
< jonasschnelli> oh
< jonasschnelli> Unexpected exception caught during testing: error('cannot add item to database',)
< wumpus> hah there goes your 1TB SSD? :-)
< jonasschnelli> Got the exception after Test 94: PASS [132]
< jonasschnelli> du -sh /var/folders/hp/kb9p9q8x4k3_z_ccy588hxrc0000gn/T/testhuweo6jb/15776
< jonasschnelli> 25M/var/folders/hp/kb9p9q8x4k3_z_ccy588hxrc0000gn/T/testhuweo6jb/15776
< jonasschnelli> 25MV
< jonasschnelli> MB
< jonasschnelli> Oh.. wait... 128GB
< jonasschnelli> Yes. Same issue. :)
< wumpus> :)
< wumpus> can you try the solution of https://github.com/bitcoin/bitcoin/issues/8605 ? e.g. replace dbm.ndbm with dbm.dumb in qa/rpc-tests/test-framework/blockstore.py
< * jonasschnelli> pulling
< jonasschnelli> running with dbm.dump now
< jonasschnelli> again 128GB
< jonasschnelli> and the execp.
< wumpus> that's very strange. dbm.dumb should be a native python implementation if dbm, not affected by the OS
< * jonasschnelli> doublechecking
< jonasschnelli> Yes. I did change to "import dbm.dumb" in qa/rpc-tests/test_framework/blockstore.py
< wumpus> note that there are three occurences of "dbm.nbdm" in that file
< jonasschnelli> ah..
< luke-jr> uh, the Python tests are trying to use BDB directly? that's unlikely to be 4.8 even on Linux, right? O.o
< wumpus> luke-jr: that doesn't matter, they're not used to communicate with bitcoind nor used for a wallet
< luke-jr> oh
< wumpus> and no, they don't request bdb, they request *a* ndbm implementation, which happens to be bdb on linux
< wumpus> but on BSDs its the libc ndm implementation, which differs
< wumpus> anyhow please read back, I feel I'm repeating myself
< wumpus> jonasschnelli: same problem on freebsd (without .dumb patch) - the file becomes huge
< jonasschnelli> 1GB now...
< jonasschnelli> (during pass 94)
< jonasschnelli> test 94
< wumpus> that's normal size
< jonasschnelli> I don't have the insight here. Is dbm BDB? Whats the difference between dumb and ndbm?
< jonasschnelli> (maybe I should read back)
< luke-jr> jonasschnelli: completely different implementation of a key-value database afaik (and different format ofc)
< wumpus> they're all key-value stores, which offer the same high-level interface but differ under the hood
< jonasschnelli> Is the dumb module much slower? If no, why not use dump for the tests?
< jonasschnelli> *why not use dumb
< wumpus> I think dumb is much slower, that is what the manual says at least
< wumpus> I haven't *tried*
< wumpus> but it seems it's the safe choice
< jonasschnelli> Theres also a gnu version...
< luke-jr> gnu is GPL, but I'm not sure we care for Python tests
< wumpus> gah, we're not going to include it in our source tree
< wumpus> although BSDs generally don't haev that installed by default
< wumpus> but dumb works fine as a fallback
< wumpus> I think we should default to the current db for Linux, and on all BSD variants use dumb
< wumpus> or use dumb everywhere if the perfrmance is acceptable for this use
< jonasschnelli> Can you easy switch between dumb/db based on the platform in python? Probably yes.
< GitHub123> [bitcoin] MarcoFalke opened pull request #8832: [rpc] throw JSONRPCError when utxo set can not be read (master...Mf1610-rpcUtxoFail) https://github.com/bitcoin/bitcoin/pull/8832
< * luke-jr> wonders how badly things would blow up if we used a map :p
< wumpus> jonasschnelli: yes, they are just modules that have the same interface
< wumpus> easy enough to import as <something>
< jonasschnelli> wumpus: re UTXO set streaming over rest
< jonasschnelli> I get an assertion:
< jonasschnelli> Assertion failed: (valid_), function key, file leveldb/db/db_iter.cc, line 67.
< wumpus> jonasschnelli: ok, maybe someting broke in the rebase, haven't looked at it much, will look at that later
< jonasschnelli> Thanks...
< pedrobranco> hi wumpus, so jonasschnelli has able to reproduce the same issue right?