< bitcoin-git>
[bitcoin] gmaxwell opened pull request #9199: Always drop the least preferred HB peer when adding a new one. (master...remove_high_bandwidth_zombies) https://github.com/bitcoin/bitcoin/pull/9199
< vanishing-coins>
hey guys, I believe I have found a rather serious issue with 0.13.1 but I am trying to figure out what is causing it
< vanishing-coins>
the symptom is that as coins are being sent through standard sendtoaddress RPC calls, overtime some change gets 'lost' until the wallet trends to 0, and the true balance is only restored when restarting Bitcoind. I am not speaking hyperbolically
< vanishing-coins>
coins are never 'lost' but bitcoind has to restart for the correct balance to be shown
< vanishing-coins>
I help multiple clients manage their nodes and this is the second time that I have seen this happen
< vanishing-coins>
both clients were running 0.12 before without issue, upgraded to 0.13.1 and HD wallets and both had the issue within 48 hours of upgrading
< vanishing-coins>
both are running on m1.smalls on Amazon EC2
< vanishing-coins>
1.7 gigs of ram, is it possible that unexpected behavior manifest due to low RAM? what would I be grepping for in the debug.log if so
< vanishing-coins>
the max memory pool is set to 300mb
< instagibbs>
vanishing-coins, do you have debug.log to share?
< instagibbs>
it sounds awfully similar to if you somehow chain too many transactions in mempool, it can throw an error and "lose" coins until restart
< vanishing-coins>
instagibbs i do
< vanishing-coins>
and that is definitely very possible -- can you elaborate on that chaining issue?
< vanishing-coins>
is that a product of having too small a memory pool? any link i can read up on?
< instagibbs>
well a look at the debug log when this happens would be helpful
< instagibbs>
no, sorry it's likely not the issue, just thinking out loud. debug.log sharing is best
< gmaxwell>
instagibbs: that wouldn't be until restart, no, it would be until after some of the chain confirms and then the retransmit logic fires.
< vanishing-coins>
hey sorry got disconnected
< vanishing-coins>
I am uploading the debug.log now
< vanishing-coins>
I also dropped memorypool size to 150 mb, and dbcache to 80
< vanishing-coins>
but very strange all around that two completely separate instances on EC2 of the same type were running fine on 0.12 but both, within 72 hours of upgrading, had the same exact issue :/
< luke-jr>
cfields: ping
< luke-jr>
should we be doing UpdateUncommittedBlockStructures in GBT proposals? :x
< cfields>
luke-jr: are gbt proposals used?
< luke-jr>
cfields: that's currently how I'm trying to test :D
< cfields>
but, as long as they're there, yea, i suppose so
< cfields>
luke-jr: which reminds me, in the same vein, i think vbrequired needs some segwit love too?
< luke-jr>
?
< cfields>
luke-jr: i just glanced at it briefly tonight, vbrequired is hard-coded to 0 as far as i can tell?
< luke-jr>
cfields: as it should be right now
< cfields>
maybe i don't really understand what it's for, i just made a note to look deeper into it later
< cfields>
(or ping you :p )
< gmaxwell>
jonasschnelli: I think the GUI should say "X blocks behind" instead of "X days/weeks/months/years" -- the latter seems to be frequently misunderstood as how long the sync is going to take. .. or perhaps we could do something smarter than that.
< luke-jr>
cfields: it's a constraint on the miner, vbs they *must* signal
< luke-jr>
gmaxwell: the master/0.14 GUI currently has a real ETA
< cfields>
luke-jr: ah, ok. I thought it was vbs they must understand. nm, then.
< luke-jr>
cfields: that's "rules", since when they must understand it, there is no bit assigned anymore ;)
< cfields>
luke-jr: yep, makes sense
< luke-jr>
huh, serializing CTransaction can modify it? :o
< cfields>
luke-jr: sorry, didn't mean to hijack. But we may as well fix up proposals if there's a use-case
< luke-jr>
(it resizes the witness vector to match the input vector)
< luke-jr>
cfields: well, I guess it's either that and/or make Eloipool submit witness-form gen tx
< wumpus>
gmaxwell: it used to say 'N blocks behind' a long time ago, was switched after lots of people complaining that blocks aren't a good indication of anything, though I'm not sure the time is that useful either...
< wumpus>
gmaxwell: there's been so much iteration on that damn progress bar at some point :p
< gmaxwell>
I was thinking that as I typed it...
< wumpus>
oh I remember now, that complaint was more like 'blocks are too technical!'
< wumpus>
'show only blocks count in the debug interface and hover' well okay
< gmaxwell>
luke-jr: I was just looking at gui in master... got 'n days behind'. which sounds like how long it will take.
< wumpus>
maybe things changed with the blockchains hype
< luke-jr>
gmaxwell: so the ETA doesn't distract enough from it, you're saying? :x
< gmaxwell>
To be honest it might as well say "working hard at catching up! here is a spinner /-\_/-\..."
< luke-jr>
lol
< gmaxwell>
luke-jr: I didn't see it! :P
< wumpus>
here's dancing hamsters...
< luke-jr>
I wasn't even looking for it, yet saw and appreciated it :p
< gmaxwell>
but I've seen two reddit threads recently where they thought the x years behind was the catchup time, and one person on IRC. ... lemme go look
< sipa>
use this early in the chain when we don't have a good estimate yet: ¯\_(ツ)_/¯
< wumpus>
yes I believe you that people get confused about it
< luke-jr>
yeah, Core 0.13 doesn't have it yet
< luke-jr>
I once had a feature request for Knots to have a button to click to play an audible "to the moon" sound bite..
< sipa>
luke-jr: fixed in 8580
< wumpus>
it's not really telling what it is doing either, e.g. if it was more apparent that it was validating history then in that context 'X years behind' makes sense
< luke-jr>
someone want to make an animation of it checking the blocks? :p
< gmaxwell>
okay the progress popup box I just dismissed. maybe now that we have that we don't need the progress bar to say 'foo behind'?
< wumpus>
then again the new modal overlay with ETA and such is really nice and mitigates these concerns
< luke-jr>
gmaxwell: I suggest change the progress bar to "Synchronising (ETA <x>)" ☺
< wumpus>
luke-jr: +1
< fanquake>
I agree, the new overlay is a good improvement.
< gmaxwell>
luke-jr: that would be fine.
< wumpus>
luke-jr: haha an animation of someone retrieving books of transactions from a chain of blocks and checking them?
< fanquake>
wumpus or gmaxwell, have you seen any Boost related memory leaks during your testing?
< gmaxwell>
the popup is kind of a sea of grey on my screen.
< fanquake>
Sorry to change topic, but I seem to have fallen down some memory leaks rabbit holes.
< gmaxwell>
fanquake: no!
< wumpus>
fanquake: nope.
< wumpus>
fanquake: qt and ancillary GUI libs (Fontconfig etc) is the only thing leaking here
< wumpus>
and all the leaks apparent in other libraries downstream seem to come from there
< fanquake>
Hmm ok you see leaks from leveldb as well?
< wumpus>
no
< luke-jr>
wumpus: I was thinking more of a horizontal line of pages sized based on the block size, and a red line scanning over them ;)
< wumpus>
nothing in the core code
< gmaxwell>
No. There is a little bit from the openssl RNG.
< wumpus>
bitcoind is 100% leak clen
< gmaxwell>
but bitcoind itself is 100% leak clean for me.
< luke-jr>
wumpus: valgrind thinks the wallet code leaks
< wumpus>
(at least the normal exit path, I haven't tried all kinds of panic shutdown scenarios ofc)
< luke-jr>
it was around some BDB mess, so I didn't look far
< luke-jr>
(one time at startup)
< wumpus>
luke-jr: not seeing that here. But I use the system berkeleydb in Ubuntu 16.04
< wumpus>
not 4.8 or that shit :)
< luke-jr>
I use system 4.8 :P
< fanquake>
Ok, I'll post a few details in a sec.
< wumpus>
I honestly don't even know what version it is... lemme check. 5.3.
< wumpus>
in any case I don't think we can ever prevent all one-time leaks in the GUI code, too many moving parts
< gmaxwell>
they're mostly interesting to fix because sometimes they're actual bugs, and keeping down the inconsequential ones avoids hiding the actual bugs.
< wumpus>
the only reason they're annoying at all is that they clutter the overview, so you may miss repeated leaks
< wumpus>
right - I found an actual (minor) bug due to it with the rpc console thread
< fanquake>
What I'm seeing seems to be Boost related, somewhere in CCoinsViewCache::FetchCoins()
< wumpus>
do you see it with only the GUI, or with bitcoind too?
< wumpus>
(not seeing them in any case but maybe an OSX thing?)
< fanquake>
Have only tested with bitcoin-qt so far, I'll look at bitcoind now.
< fanquake>
Here's a stack trace for the Boost leak, http://pastebin.com/V2QY8NT4 , not seeing anything on bitcoind so far (other than something related to berkely-db)
< fanquake>
That is without shutting down, just while it's running.
< jonasschnelli>
fanquake: I'm also compiling against Qt 5.7
< wumpus>
you can only see *leaks* after you've shut down
< jonasschnelli>
though I don't see the CCoinsViewCache::FetchCoins() leak
< wumpus>
profilers can't predict the future of what an application is going to do with some memory, so you can't be sure some memory won't be released until the end of the shutdown sequence
< jonasschnelli>
wumpus: I guess some leaks are detectable during runtime (when all pointers have lost the object)... not?
< wumpus>
(although it could use some heuristic like "does the application retain any pointers to it" but that's horribly imprecise at most for C++)
< wumpus>
jonasschnelli: not in general for unmanaged languages
< wumpus>
and managed languages tend to have garbage collectors anyway :-)
< jonasschnelli>
I guess it often leads to wrong detected leaks... (maybe losts).
< luke-jr>
wumpus: I think it can be done with sufficient accuracy that the alternative is undefined behaviour?
< luke-jr>
or at least very bad code :p
< jonasschnelli>
I also think the Leak detector fanquake and I are using (Apples "Instruments") are not really made for C++... It's perfect when analyzing objective-C code.
< * luke-jr>
actually finds runtime leak detection more useful than after exit, since many things use data but don't clean it up as well
< fanquake>
Ok, I'm probably using the wrong terminology here then. I don't have too much experience using these kind of tools.
< wumpus>
luke-jr: well that depends on your definition of 'bad code' I suppose, I'm just talking about theoretic properties not judging any particular use of cpu cycles :p
< fanquake>
jonas Yes, hat might also be a part of the problem.
< luke-jr>
wumpus: by bad code, I mean serializing the pointer in a uint8_t array or something
< wumpus>
luke-jr: yes but I don't think e.g. pointer bit flipping tricks are by definition bad code, it just depends on the context, the resources available etc
< jonasschnelli>
IMO all the CDB (BDB) leaks reported by "Instruments" are wrong reports...
< fanquake>
jonasschnelli so when we're looking at anything CWallet related in instruments (which is all I see looking at bitcoind) what are we seeing exactly?
< fanquake>
heh, just what I was wondering.
< jonasschnelli>
fanquake: tons of BDB leaks...
< jonasschnelli>
I guess >10MB
< jonasschnelli>
(for a 300tx regtext wallet)
< wumpus>
PSA: the macosx tools are wasting your time
< wumpus>
:-)
< fanquake>
:o
< jonasschnelli>
wumpus: there is also the time profiler in "Instrumenst"... this is really cool.
< jonasschnelli>
gpref like
< jonasschnelli>
gperf
< jonasschnelli>
wumpus: how did you detected the Qt leaks? IMO last time I used it on Bitcoin-Qt on Ubuntu I had to force shutdown my machine.. :)
< wumpus>
luke-jr: yes - LLVM sanitizers that's what I used in #9190, built from latest LLVM git even, I have to track that anyhow for some GPU stuff
< fanquake>
So we can ignore all of the certificate related Qt stuff as well?
< wumpus>
it generates really nice rainbow console output, makes finding a use-after-free bug almost enjoyable
< wumpus>
fanquake: not sure about ignoring anything, but please if you report leaks make sure that you are talking about the allocation residue after quitting the application
< fanquake>
wumpus: Yep, have a bit better handle on this now. Looks like using some tools other than the native OS X stuff could be the way to go.
< wumpus>
jonasschnelli: but yes visual profilers can be really cool, at a previous employer for solaris we had one that showed exactly what an application was doing on each core in a timeline overview, that was really useful for checking efficiency of concurrency etc
< wumpus>
I kind of miss that in the linux toolset but I may be missing something, so many different tools and instrumentations
< sipa>
dtrace?
< wumpus>
yes I think it was based on that
< wumpus>
but without having to actaully write dtrace scripts :)
< jonasschnelli>
wumpus: when compiling with CPPFLAGS/LDFLAGS +"-fsanitize=address", I get Undefined symbols during liking (___asan_stack_malloc_5, etc.)? Any ideas?
< wumpus>
jonasschnelli: it should try to link against the asan runtime library
< jonasschnelli>
I passed -fasnitize=address as LDFLAGS ... hmm...
< jonasschnelli>
*-fsanitize=address
< wumpus>
things like "libclang_rt.asan_cxx-i686.a"
< wumpus>
I had to put compiler-rt in my llvm/clang build to get them in the first place, but I'd expect distro-build versions to have that enabled by default
< wumpus>
in any case yes passing that to the linker should try to link them automatically
< jonasschnelli>
hmm... libclang_rt.asan_osx_dynamic is present...
< jonasschnelli>
I guess OSX XCode clang uses a different linking approach in oder to support ASAN for all its supported platforms (iOS/ apple tv/ apple watch, etc.).
< jonasschnelli>
Probably better to use a self-compiled clang (or though homebrew)
< fanquake>
I know we already need documentation for lots of existing stuff, but this kind of debugging/configuration/dev tool type stuff would also be worth documenting. At least more than what we have now.
< fanquake>
It'd be nice to have *lots* of people running the identical binary, and similar tools.
< jonasschnelli>
I tried to add the ASAN to the cofigure.ac
< wumpus>
agree that tooling documentation would be useful, though none of this is specific to bitcoin core
< wumpus>
some projects have a wiki with this kind of auxiliary documentation
< fanquake>
Yes I'm assuming there must be similar projects, maybe something like Tor, that would already have alot of this kind of stuff written/documented somewhere.
< paveljanik>
fanquake, the documentation is generic for debugging tools...
< paveljanik>
and there already is a lot of such documentation.
< * wumpus>
is trying to add CPU cycles measure to the bench framework and probably stumbled on a subtle counting bug w/ rescaling the iteration count
< wumpus>
this can cause especially the 'min' time to be off
< gmaxwell>
wumpus: darn. I thought I'd tested that fairly well.
< bitcoin-git>
[bitcoin] laanwj opened pull request #9200: bench: Fix subtle counting issue when rescaling iteration count (master...2016_11_bench_fix) https://github.com/bitcoin/bitcoin/pull/9200
< gmaxwell>
oh pfft.
< wumpus>
yea :)
< gmaxwell>
wumpus: I guess I'd reasoned it would be 0 mod n due to the initial count, oh well.
< wumpus>
yes it's really subtle, I only discovered it because I was printing lots of debug info to debug the cycle counter, then saw the first run after a scale was off
< wumpus>
the use of floating point arithmetic in the benchmarking loop isn't too great either but I'll leave that for another time
< gmaxwell>
yea, esp on the kind of platforms that you really need to benchmark things on.
< gmaxwell>
I wanted to eliminate that too when I last touched it but I thought it would be too much at once. :)
< wumpus>
exactly
< jouke>
re IRC meeting: we use the account system
< wumpus>
yes there needs to be an explanation how to do that without accounts / with the new labels API
< jouke>
Not something we can't do ourselves, but it was there, so why not use it :P
< wumpus>
and for that specific use case you don't really need accounts, just a way to track/group incoming payments using a label
< jouke>
And keep track of reorgs?
< jouke>
Again, not something we can't do ourselves and we've known that accounts are deprecated for a while, but I was reading the IRC meeting summary and someone was wondering if anyone talking to the devs was using accounts, so I started talking :P
< Victorsueca>
the whole accounts thing doesn't make a lot of sense
< jouke>
What do you mean Victorsueca?
< Victorsueca>
I received some bitcoins to a address labeled "Victor" and later spent them all
< Victorsueca>
now the account "Victor" still has the balance and the default "" account is in negative
< Victorsueca>
and I didn't use the move command
< jouke>
Victorsueca: because you didn't use "senfrom"?
< Victorsueca>
nope, didn't use that either
< thermoman>
sdaftuar: we do import privkeys every 2 or 3 days
< jouke>
That's why it's negative :).
< jouke>
Victorsueca: but I get why it's confusing, that's one of the reasons it's deprecated :)
< Victorsueca>
is it safe if I move all from bitcoin to "" or I would mess with the wallet balance calculation?
< Victorsueca>
from Victor*
< jouke>
Victorsueca: sendtoaddress only subtracts the amount from the default account, sendfrom subtracts the amount from a specific account. Victorsueca it's safe, but again, don't start using accounts, it will be removed in future versions.
< Victorsueca>
ahh nice
< luke-jr>
Victorsueca: it makes sense.
< luke-jr>
jouke: unless you're willing to *maintain* the account system, your usage isn't likely to change its removal
< luke-jr>
jouke: and even if you're willing to maintain it, I think the conclusion will be "please do it outside bitcoind"
< jouke>
luke-jr: I know.
< Victorsueca>
luke-jr: btw, I was wondering when will you be able to push segwit into bfgminer
< luke-jr>
Victorsueca: you mean libblkmaker :P
< luke-jr>
Victorsueca: testing would be helpful
< Victorsueca>
luke-jr: I would, but my Antminer U3 seems to only work on windows and I didn't find any instructions on how to cross-compile
< luke-jr>
Victorsueca: p.s. if you want to mine testnet with it, tmp.pool.bitcoin.dashjr.org port 3334 (stratum) or 8337 (GBT & getwork)
< luke-jr>
Victorsueca: no promises I don't kill it in a few hours tho, it's eating CPU time
< luke-jr>
besides, better to solo mine :P
< wumpus>
jouke: no, with labels instead of accounts you don't have to manually keep track of reorgs, you can still check the total balance received to a label, for example. You just cannot do accounting, so move coins instead of addresses from one label to another
< Victorsueca>
luke-jr: thanks, I will try to solo-mine for a while too to see if it signals segwit properly
< bitcoin-git>
[bitcoin] laanwj opened pull request #9202: bench: Add support for measuring CPU cycles (master...2016_11_bench_cpu_cycles) https://github.com/bitcoin/bitcoin/pull/9202
< Victorsueca>
luke-jr: any tip on how to detect my Antminer U3? I've been following the instructions on README.ASIC ut it won't detect it
< Victorsueca>
but*
< luke-jr>
Victorsueca: the official drivers are installed? (not WinUSB/Zadig)
< Victorsueca>
ahh, It needs to be with the default driver?
< Victorsueca>
I have WinUSB
< luke-jr>
yes
< Victorsueca>
luke-jr: still not detecting it
< luke-jr>
with the options in README.ASIC?
< Victorsueca>
yep, and with the default sillabs driver
< wumpus>
this is not related to bitcoin core development, can you move this to #bitcoin?
< Victorsueca>
sure
< bitcoin-git>
[bitcoin] jnewbery opened pull request #9203: [trivial] Fixes the RPC help text for waitforblockheight (master...waitforblockheightcomment) https://github.com/bitcoin/bitcoin/pull/9203
< bitcoin-git>
[bitcoin] jnewbery closed pull request #9203: [trivial] Fixes the RPC help text for waitforblockheight (master...waitforblockheightcomment) https://github.com/bitcoin/bitcoin/pull/9203
< instagibbs>
;;later tell vanishing-coins Looks like you are trying to make dust, aka too-small outputs, and your wallet is refusing to send them
< bitcoin-git>
bitcoin/master 1260c11 Pavel Janík: Mention the new network toggle functionality in the tooltip.
< bitcoin-git>
bitcoin/master ac489b2 Jonas Schnelli: Merge #9130: Mention the new network toggle functionality in the tooltip....
< bitcoin-git>
[bitcoin] jonasschnelli closed pull request #9130: Mention the new network toggle functionality in the tooltip. (master...20161111_disable_network_tooltip) https://github.com/bitcoin/bitcoin/pull/9130
< nsh>
toggles all network activity?
< nsh>
handy i guess
< jonasschnelli>
nsh: The PR above is just a cosmetic one.
< jonasschnelli>
The actual toggle It's in master since some days
< jonasschnelli>
Just press the network-icon on the top right in the status bar.
< adiabat>
howdy, I've having trouble getting 0.13.1 synced up to testnet3
< adiabat>
running on a raspi2
< adiabat>
got a "EXCEPTION: St9bad_alloc" in debug.log
< sipa>
how much ram does that have?
< adiabat>
1G
< adiabat>
I've gotten that exception in different places when deleting and starting over
< sipa>
hmm, maybe you need a bit of tweaking to reduce dbcache/mempool, and limit the number of peers?
< sipa>
it shouldn't be much
< adiabat>
this is all with 1 peer via connect= in the conf
< adiabat>
also I'm running txindex=1
< sipa>
ah
< adiabat>
it was working OK on 0.13.1 but that had been upgraded many times. Then I broke my microSD card in half :(
< adiabat>
IBD from scratch on .13.1 on the pi may need different settings?
< sipa>
maybe... it clearly means OOM
< adiabat>
I'll try dbcache=64
< adiabat>
If this works (will know in ~24hrs) maybe we can put a readme or something. With the arm binaries I think more people will try running on a pi2/pi3
< adiabat>
also far be it from me to complain but trashing the who DB requiring re-index on an OOM error is... unfortunate.
< adiabat>
(Not offering to fix that! heh)
< adiabat>
who = *whole
< sipa>
we should work on being able to dump the utxo set to a file and load it back
< adiabat>
I've never had bitcoind get in trouble with RAM on a regular x86 computer, and I've run it on PCs with 2G of RAM
< gmaxwell>
2GB is a lot more than 1GB. :)
< adiabat>
heh... I wonder what the least powerful full node is. raspi is getting close maybe, but I'm sure someone out there has it running with 512MB
< Victorsueca>
adiabat: just download more ram ;)
< sipa>
it's not cisco hardware
< pigeons>
ramdoubler.exe
< Chris_Stewart_5>
Is the serialization format described in BIP141 only used when calculating a wtxid, or is this serialization format also used when propogating a tx across the p2p network?
< jtimon>
also if anyone is bored, you can fetch #8994 , grep and remove 'self.chain = "regtest"' and you should have 4 tests failing, ie p2p-comapctblocks.py, segwit.py, mempool_packages.py (extended) and pruning.py (extended)