< gmaxwell>
the leveldb logging with debug=1 is obnoxious, now I see why jnewbery wanted the exclude.
< gmaxwell>
wumpus: uh. so boolean operations don't appear to exist for scoped enums. so ... I don't think I can use them without defining a set of boolean operations for them.
< sipa>
gmaxwell: eh?
< sipa>
ah, scoped enums don't have an implicit conversion to int
< sipa>
you can still explicitly convert them, though
< sipa>
but enums are generally a bad approach for bitfields, as they're intended to be exhaustive
< gmaxwell>
sipa: wumpus suggested that I change the log print thing to use a scoped enum. The original thing that inspired this change is that rebroad wanted to do some horrible hack to do the logical equivilent of LogPrint(LOG_NET | LOG_MEMPOOL, ...). (and of course the matching also uses it as a bitfield.)
< sipa>
right, a normal enum (which behaves much more like an int) may be more appropriate
< gmaxwell>
well thats what I implemented.
< sipa>
awesöme
< gmaxwell>
in any case. I'm happy to change it to whatever. someone who has an opinion needs to tell me what to do. I think I only need a couple casts to make the enum class stuff go, but it would be ugly to do "LogPrint(LOG_NET | LOG_MEMPOOL," later unless we provide | for the type.
< bitcoin-git>
bitcoin/master 625488a Wladimir J. van der Laan: util: Work around (virtual) memory exhaustion on 32-bit w/ glibc...
< bitcoin-git>
bitcoin/master ba12b3a Wladimir J. van der Laan: Merge #10120: util: Work around (virtual) memory exhaustion on 32-bit w/ glibc...
< bitcoin-git>
[bitcoin] laanwj closed pull request #10120: util: Work around (virtual) memory exhaustion on 32-bit w/ glibc (master...2017_03_address_space_exhaustion_workaround) https://github.com/bitcoin/bitcoin/pull/10120
< sipa>
wumpus: have we ever done benchmarks for the performamce impact of having only 1 (or 2) arenas?
< wumpus>
sipa: afaik, no. I've only done that on systems that would crash otherwise
< wumpus>
so no way to compare :)
< sipa>
infinite speedup, on those systems!
< wumpus>
exactly!
< gmaxwell>
for 32bit I don't think we really have a choice, in any case.
< wumpus>
yes. the question is do we also want to customize arenas on 64-bit
< gmaxwell>
sipa pointed out today that we really need to test that with network syncs since a chainstate reindex isn't really multithreaded in terms of malloc activity.
< wumpus>
though I don't think so, I think on the longer run we should address leveldb's mismanagement of memory instead, as it's the largest reason we can't get along with many arenas
< gmaxwell>
Also I made the observation that the mempool on disk likely increased our memory usage in the presence of multiple areans, because instead of only having two threads with substantial allocations it resulted in three.
< wumpus>
yes
< gmaxwell>
earlier I proposed a new database durability strategy to pieter that, if it works, will let us radically reduce peak memory usage.
< wumpus>
and yes it may very well be that arenas doesn't help at all for our use case, but yes, profiling would make sense
< gmaxwell>
(by elimiating large batch writes)
< wumpus>
the glibc allocator also has some other 'tunables' that may have an effect on performance
< morcos>
wumpus: ok take a look at that. This would be my recommendation, but I broke it up into commits so we can talk about what makes sense for 0.14.1 and what doesn't....
< morcos>
I care primarily about the first commit
< morcos>
sipa: i was trying to reason about whether we need the 2x factor in the rollback test
< morcos>
i went back and forth.
< morcos>
is it true that non-dirty entries in the pcoinstip aren't subject to a 2x usage on flush?
< morcos>
if so then i guess i agree...
< sipa>
morcos: right
< morcos>
but certainly if pcoinstip grows which it will in the rollback, and then it will be flushed later...
< sipa>
are you sure? i thought we changed that test to not use the main cache
< morcos>
pcoinstip will grow, but will only grow with non-dirty entries
< morcos>
for instance if your tip is block 100, you flushed after block 99 and you are rolling back block 99, won't pcoinstip get the outputs created in block 99 pulled into it
< morcos>
which will be marked dirty in coins, but never flushed to pcoinstip
< sipa>
morcos: pcoinstip is not involved in the rollback test at all
< sipa>
it uses a private cache on top of pcoinsdb
< morcos>
oh... i missed that
< sipa>
if (!CVerifyDB().VerifyDB(chainparams, pcoinsdbview, GetArg("-checklevel", DEFAULT_CHECKLEVEL),
< morcos>
i assumed coins was built on top of pcoinstip
< sipa>
it used to be, but it caused a memory blowup that quickly caused a flush afterwards anyway
< morcos>
ok, then i agree completely unnecessary there... will remove
< morcos>
while we're at it.. are you ok with the other changes.. i might as well get it to mergeable state.. i agree this is all relatively minor
< morcos>
oh nm, maybe its fine.. i was just confusing myself with whether the 90% is still the right number, but you are comparing the doubled calculation with the 90% number, so it is fine
< bitcoin-git>
[bitcoin] sdaftuar opened pull request #10134: [qa] Fixes segwit block relay test after inv-direct-fetch was disabled (master...2017-03-fix-segwit-relay-test) https://github.com/bitcoin/bitcoin/pull/10134
< bitcoin-git>
[bitcoin] jnewbery opened pull request #10135: Send the correct error code in reject messages (master...correctrejectmessages) https://github.com/bitcoin/bitcoin/pull/10135
< bitcoin-git>
[bitcoin] jnewbery closed pull request #10123: Allow debug logs to be excluded from specified component (master...debugexclude) https://github.com/bitcoin/bitcoin/pull/10123