< xRavenheart>
anyone able to assist me in my queries about a school project please?
< Randolf>
xRavenheart: What's your school project about? Bitcoin development?
< xRavenheart>
hi anyone able to assist me in a school project related question?
< sipa>
02:47:32 < Randolf> xRavenheart: What's your school project about? Bitcoin development?
< luke-jr>
xRavenheart: not here
< Randolf>
luke-jr: You're right. However, they just contacted me via PM in response to my response, so they probably have a secondary account watching this channel (I suspect).
< Dizzle>
Randolf: he's here, rejoined. I think luke-jr was just suggesting he ask elsewhere, e.g. #bitcoin
< sipa>
i'm talking in PM to him
< xRavenheart>
Ok guys, it seems like i should raise the question here instead of PM-ing
< xRavenheart>
Basically, im a final year student doing FYP and my project scope is to modify on an existing lightweight client by bringing in optimizations from other clients (full/lightweight).
< Randolf>
Dizzle: Ah, yes. I suggested that asking here may be appropriate since it might involve core development. I also suggested that they ask in the #bitcoin and ##altcoins channel.
< xRavenheart>
This optimization may be in any form such as block headers download, transacting speed, retrieving of wallet addresses, etc etc.
< sipa>
xRavenheart: you're welcome here to ask questions about bitcoin's design and operation
< xRavenheart>
E.g. For instance Bitcoin Core has optimizations 1,2 and Electrum has optimizations 3,4
< xRavenheart>
I'm required to bring together these optimizations into one application
< sipa>
though as i told, most of that will not be applicable to other types of clients
< xRavenheart>
Is anyone able to first provide me with a clear architecture diagram of bitcoin?
< xRavenheart>
I've been trying to understand the system but it is way too broad and i have no clear idea of where to start
< sipa>
ii don't think that exists
< achow101>
is there any particular reason that Core is using 26 GB of RAM during a reindex?
< achow101>
(out of 32 available on my machine)
< gmaxwell>
sipa: ^
< achow101>
my dbcache is only set to 6000
< gmaxwell>
achow101: master, right?
< achow101>
gmaxwell: yeah
< gmaxwell>
achow101: sipa saw this on his odroid too, thought it was a bug in master, but I think he had some reproduction problems.
< gmaxwell>
It may be a leak that depends on particular dependency versions.
< achow101>
I can consistently reproduce it by starting it, letting it go, and then waiting for my computer to freeze up.
< achow101>
then reboot and repeat :/
< gmaxwell>
any chance you could run in valgrind, let it obviously bloat up a bit then issue a clean stop? if it's a true leak valgrind will report it.
< achow101>
Yeah, I'll do that once I reboot it again
< gmaxwell>
(I'd bet it's not actually a true leak... we have far too little manual memory management for that to be likely... but worth checking)
< Randolf>
I could try running bitcoind to confirm memory usage too -- I've got bitcoind v0.15.1 on Windows 10 (64-bit) handy.
< Randolf>
Which command-line parameters are you using?
< gmaxwell>
0.15.1 doesn't exhibit this behavior in any testing we've seen so far.
< Randolf>
Oh. Which version are you using?
< gmaxwell>
git master... the current in-development stuff.
< Randolf>
Oh. I see.
< achow101>
Some build of master from sometime in the last few days
< achow101>
This is fairly recent, I think it's from yesterday or the day before
< achow101>
Although maybe I only noticed it now because I had to reindex
< Randolf>
Which OS are you running on?
< achow101>
Ubuntu 17.10
< Randolf>
Oh, I don't have that available. I have a NetBSD server I could run it on if you need an extra test run.
< Randolf>
I'd have to get sync'd though because I don't have bitcoind running there now. Maybe I should see if I can get it going and get the sync started for now.
< achow101>
also, i'm running bitcoin-qt right now, not bitcoind
< Randolf>
These servers don't have any GUI stuff installed. So I won't be able to try bitcoin-qt on them. But I could at least try bitcoin-qt on Windows 10 to confirm whether the same problem occurs cross-platform.
< achow101>
btw I'm using commit 00d25e90db06149fa456b0a8f15b7b68005ff9c5 which is the current head of master
< achow101>
gmaxwell, sipa: when did you observe this issue on the odroid?
< Randolf>
Hmm, I'm running into problems getting bitcoind compiled on NetBSD, but I've had problems getting other things to compile due to a learning curve on my part so I'll have to work on this. In the meantime, I'll look into commit 00d25e90db06149fa456b0a8f15b7b68005ff9c5 (as I continue to learn more
< achow101>
it might be useful to walk back through the commit history testing each merge until we get to one that doesn't exhibit the leak. that would then tell us what commit introduced the problem, in theory.
< achow101>
although that will be very tedious to do
< sipa>
achow101: i have a difficult time reproducing it; i just saw it twice
< Randolf>
about GitHub) to see if there's something in there that I can test out on Windows.
< sipa>
it would be asome if you can bisect
< achow101>
sipa: I'm going to try that once I finish this valgrind test
< sipa>
great
< achow101>
but because it's valgrind, it's going slowly
< sipa>
i do know 0.15.1 is unaffected
< achow101>
will bisect work with 0.15.1 even though it is not part of master's commit history
< sipa>
if not, you can always use the last common forking point between master and 0.15.1 is goof
< achow101>
indeed
< sipa>
*good, not goof
< Varunram>
achow101: If you set the dbcache in the GUI, you might be affected by this: #11788
< achow101>
I don't use the gui for any settings at all
< Varunram>
phew, along those messages, I read somewhere you were running qt, so..
< achow101>
I do run qt for testing things, and I noticed this issue while running qt
< sipa>
that's very unlikely to be related
< achow101>
but I'm just running bitcoind in valgrind right now
< Varunram>
yep, mostly not :) just wondered
< achow101>
damn valgrind is slow
< eck>
should be about 10x slower
< sipa>
much worse for cryptographic code
< achow101>
It's still in the rescanning part of init
< eck>
the actual numbers depend on how many memory accesses you are doing, all of the instructions that hit main memory are "emulated"
< sipa>
all instructions are emulated
< sipa>
it's not like it's running in a VM
< sipa>
but memory access get additional instrumentation
< eck>
i thought it worked by rewriting all of the instructions that access main memory, and then sort of fusing the non-memory instructions, i read julian seward's paper on it a long time ago
< eck>
but i certainly could be wrong
< sipa>
hmm, so could i
< sipa>
i didn't know it was that advanced, actually but it certainly could be
< eck>
in some cases it has to emulate cpu registers, in other cases the emulated cpu register may be the actual host cpu registers, section 3.4 http://www.valgrind.org/docs/valgrind2007.pdf
< eck>
clearly if the registers need to be emulated the overhead will be quite high
< Randolf>
Well, I was unsuccessful at getting bitcoind to compile on NetBSD because of problems with Berkeley DB API function naming, and Boost incompatibility problems. I'll have to try again when get some more time (and I will try again).
< eck>
fyi if you run configure with --disable-wallet you don't need berkeleydb, which would just leave you to figure out boost
< achow101>
I gave up on valgrind. It was doing the rescanning thing and moving at one percent per hour. I'm going to do the git bisect stuff tomorrow
< eck>
were you using massif or memcheck?
< achow101>
memcheck
< eck>
massif will just straight up tell you at any given point of time, where all the allocations came from
< eck>
which is useful if you can't wait for the program to run to completion
< eck>
iirc it's even slower than memcheck though
< achow101>
the problem was that it wasn't getting passed the init part. the leak is definitely coming from somwehere past the init part
< eck>
i have heard that ASAN is faster than valgrind but i haven't used it
< Randolf>
eck: Thanks. There's an incompatibility because I have the wrong version of Boost.
< eck>
it is pretty straightforward to build, if you have the patience
< Randolf>
eck: I have plenty of patience, but my problem is meeting other deadlines at the moment so I've put this on hold for now. I also need to learn more about compiling.
< achow101>
I fairly quick and rather unscientific git bisect says that e545dedf72bff2bd41c93c93eb576929fce37112 is the commit that introduces the problem.
< achow101>
so if you want to investigate, that commit and commits in its vicinity should be looked at. But I need to sleep, so I'll look at it tomorrow.
< mmhhtt>
Hello, Is there anyone knows to how can build a exchange market for other country? Is there any request from bitcoin company to me for making exc market?
< mmhhtt>
also how can work payment for buy/sell btc for other country local money?
< sipa>
achow101: that's what i feared...
< RK_>
Hello
< RK_>
I want to know about bitcoin
< mmhhtt>
#bitcoin-core-dev
< mmhhtt>
hello
< mmhhtt>
Is there any one?
< notabot_>
mmhhtt: try #bitcoin channel or reddit
< notabot_>
RK_: bitcoin.org is the best place to start.
< GAit>
I managed to compile core for android with ndk by the way, which means i don't need the glibc hack + ld library path with archlinux/debian glibc
< GAit>
thanks sipa & achow101
< sipa>
GAit: cool!
< achow101>
BlueMatt: running your patch now. what exactly do you want?
< BlueMatt>
achow101: looking for a pattern of debug.log entries like a bunch of "X event queued!" but not (enough) corresponding "X event executing..."
< BlueMatt>
s
< BlueMatt>
(or just paste debug log
< achow101>
eyballing this debug.log tail looks like each one is properly paired
< BlueMatt>
hmmm, ok, so not the issue i assumed :(
< BlueMatt>
but the memory is growing?
< achow101>
yes
< BlueMatt>
hmmmm
< achow101>
not as quickly as yesterday since I dropped the dbcache, but growing much faster than it did with 0.15.1
< BlueMatt>
ok, then its news to me, have you checked a heap profile?
< achow101>
no
< BlueMatt>
well i guess thats next? (did valgrind give you any actual leaks?)
< BlueMatt>
i assuming circular shared_ptr refs
< achow101>
valgrind... died
< achow101>
(I killed it)
< BlueMatt>
ah :/
< achow101>
but while it was running it did not output anything
< achow101>
I'll try it again and be more patient this time
< ryanofsky>
it's pretty use address sanitizer too. just: "./configure CXX=clang++ CXXFLAGS=-fsanitize=address LDFLAGS=-fsanitize=address" and run normally. it prints leaks at the end
< BlueMatt>
ugh, fuck you cfields, you sent me down a rabbit hole to replace cvBlockChanged with CValidationInterface, but I think I decided that has to wait on replacing the signals backend with our own thing
< BlueMatt>
cause you really end up wanting the validation interface stuff to have eg two threads where the ordering guarantees are only made per-client, and each new client that gets connected only receives events generated *after* when it was connected
< cfields>
BlueMatt: heh, sorry
< BlueMatt>
(pretty easy to do, I think, you just make events eg a struct and put shared_ptrs to it in each listener's queue)
< BlueMatt>
but its way more work than just fixing a fucking missing cs_main
< BlueMatt>
and is gonna have to wait for 20 other higher-priority things
< achow101>
ryanofsky: will try that at some point
< BlueMatt>
the cvBlockChanged/csBestBlock garbage is....garbage now that we have "real interfaces" to get notifications of validation events
< cfields>
BlueMatt: fwiw, i was picturing just adding a cached uint256 and a function to update it rather than notifying the cv directly
< BlueMatt>
i know you were, but lets *please* not add more in-thread callbacks out of validation :(
< BlueMatt>
I've been working too hard to get rid of those things
< cfields>
i understand your point about using our interfaces. but weighing quick fix against quick fix, that seems better to me
< BlueMatt>
meh, its a short cs_main, why not just take the cs_main
< cfields>
i'm worried that we'll end up locked there for milliseconds where we wouldn't have otherwise. Granted, the re-locking of cs_main at the bottom kinda nullifies that argument.
< BlueMatt>
yea, we re-lock anyway
< BlueMatt>
and, more importantly no one uses longpolling anyway
< BlueMatt>
afaiu
< BlueMatt>
achow101: you said you can repro by just -reindex'ing?
< achow101>
BlueMatt: yes
< BlueMatt>
concept review on #11799 would be appreciated - ignore the code thats there, its useless, but the question is do we want negative-lock-annotations (ie you may not hold cs_main when calling function x) when they are only enforced if the lock is taken directly in the calling function (unlike the requires-lock annotations, these ones dont require you pollute your entire codebase to fully document locking states)
< gribble>
https://github.com/bitcoin/bitcoin/issues/11799 | wallet: Add compile-time checking of (non-)locking assumptions for BlockUntilSyncedToCurrentChain() [wip] by practicalswift · Pull Request #11799 · bitcoin/bitcoin · GitHub
< BlueMatt>
someone want to close #11823 as NOTABUG
< bitcoin-git>
[bitcoin] practicalswift closed pull request #11799: wallet: Add compile-time checking of (non-)locking assumptions for BlockUntilSyncedToCurrentChain() [wip] (master...BlockUntilSyncedToCurrentChain-compile-time-warnings) https://github.com/bitcoin/bitcoin/pull/11799
< achow101>
hmm. valgrind didn't turn up with anything too frightetning, nor did the address sanitizer
< achow101>
only 160 bytes leaked (granted I only ran then for ~10 minutes each)
< BlueMatt>
achow101: nvm, i got it
< achow101>
.. ok?
< achow101>
BlueMatt: got it as in reproduced or found the leak?
< BlueMatt>
working on fix
< BlueMatt>
sec
< BlueMatt>
achow101: heh, well maybe I didnt have it, can repro and ran into a seemingly-related bug, however
< BlueMatt>
achow101: nope, nvm, my first guess was right
< sipa>
BlueMatt: care to elaborate?
< BlueMatt>
validation running ahead of validationinterface, you end up with a deep queue of shared_ptr<CBlock>s of all the shit you connected
< BlueMatt>
not normally an issue, but reindex is particularly easy to hit there
< sipa>
i would guess the solution is to block progress if the queue gets too long?
< BlueMatt>
indeed
< achow101>
BlueMatt: oh, I see now. so that's probably why valgrind and address sanitizer wasn't catching it
< BlueMatt>
its not a leak, thats why :p
< BlueMatt>
just an unbounded queue
< achow101>
was my git bisect correct?
< BlueMatt>
yes
< achow101>
:D
< cfields>
BlueMatt: Hmm. doesn't that imply that at the time something (say wallet) is running a callback, chainActive's tip could've progressed significantly past the payload block?
< BlueMatt>
cfields: yes, but that was always assumed to be allowed?
< BlueMatt>
cfields: heh, you're gonna murder me...the "quick fix" for this requires a net_processing-specific CNode boolean to be added :/
< cfields>
BlueMatt: how would that have manifested before moving to the scheduler thread?
< BlueMatt>
well, the "super quick fix" is to add a bunch of std::this_thread::yield()s, butttt....
< BlueMatt>
cfields: it wouldnt have, cause the calls are all on the thread that is generating the events
< cfields>
BlueMatt: right. I'm trying to reconsile that with "yes, but that was always assumed to be allowed?"
< BlueMatt>
oh, sorry, yes, just meant that that is the design of validationinterface now
< BlueMatt>
ie shouldnt be news if you reviewed the last 5 prs :p
< cfields>
well i didn't have this issue in mind, as I was still thinking too serially. Now I need to go re-review :(
< BlueMatt>
heh, sorry
< cfields>
*synchronously
< cfields>
BlueMatt: see: SyncTransaction -> AddToWalletIfInvolvingMe -> MarkConflicted. That ends up testing against chainActive for conflicts rather than the view from the callback :(
< BlueMatt>
cfields: heh, I know :(
< BlueMatt>
oh, wait, sec
< BlueMatt>
missed your point, sorry
< cfields>
BlueMatt: assuming the callbacks are lagging and chainActive is significantly ahead, MarkConflicted could be looking for conflicts on some future (or reorg'd) chain
< cfields>
(i'm just looking around for things that could accidentally be using chainActive on the callback thread)
< jamesob>
cfields: you're looking into the shared_ptr<CBlock> backup?
< cfields>
just having another look over the change that moves the validatioininterface callbacks to the scheduler thread. I guess that's what you mean?
< jamesob>
yeah
< jamesob>
didn't know if there was a particular fix in mind for that yet
< sipa>
BlueMatt, cfields: if the wallet notifications can lag behind chainActive, the wallet needs its own "tip" to test against, no?
< BlueMatt>
cfields: sorry, was just finishing the queue-drain branch...I do not believe that to be an issue? Specifically, in the conflicted case you mention here, the only way to hit it is if the wallet lags behind and the block that made a transaction conflict is already reorg'd off the main chain, in which case you dont want to mark it conflicted, and if its still gonna be conflicted once you get to the best chain a later callback will mark
< BlueMatt>
it conflicted appropriately
< BlueMatt>
jamesob: yes, I have a branch
< BlueMatt>
sipa: I do not believe that to be the case (currently), please prove me wrong!
< achow101>
BlueMatt: cfields: are you working on the fix for the queue thing or something else?
< BlueMatt>
achow101: yes
< BlueMatt>
though cfields is trying to find other issues in the original pr
< achow101>
ok
< sipa>
BlueMatt: it certainly sounds safer...
< cfields>
sipa: agreed
< BlueMatt>
sipa: I agree, but to do that you'd rewrite about another 10 functions in the wallet and probably end up refactoring half of it
< BlueMatt>
(again, please prove me wrong :p)
< cfields>
even if the conflicts aren't an issue, i still really don't like that we're testing against some seemingly random chain
< BlueMatt>
well in that one specific case its fine (and kinda the way the function was written to begin with, strangely)
< sipa>
BlueMatt: hmm, really? I would just replace every call to chainActive in the wallet with a local variable, which gets updated through the callbac
< cfields>
BlueMatt: at least for the stuff affected by callbacks, can't we just pass a pindex around instead?
< BlueMatt>
cfields: isnt that like the one place that matters from the callbacks (its been like 8 months since I've looked though)
< cfields>
BlueMatt: unsure. I'll take a closer look before commenting further
< BlueMatt>
sipa: the number of places I'd have to go read to make sure GetDepthInMainChain being disconnected from main chain is fine to be convinced of that is....high
< BlueMatt>
sipa: though I'm sure ryanofsky would appreciate it greatly
< BlueMatt>
sipa: the obvious issue is that gui would have to be aware of the wallet chain state
< sipa>
right
< sipa>
and if there's anything that looks at pcoinsTip from the wallet (is there?) that would also be affected
< BlueMatt>
I dont think there is
< BlueMatt>
sipa: ReacceptWalletTransactions/RelayWalletTransactions would need a only-if-caught-up-to-main-chain condition, but thats the only obvious ones in wallet.cpp...it may be much easier than I thought, but I already have too many open prs to want to do it any time soon