<PaperSword>
Has there been a dbwrapper written for either db or a pull?
<sipa>
not as far as i know
<sipa>
it's worth experimenting with, but it's also good to keep in mind that the database layer is effectively consensus critical, and potential bugs in it can have very severe consequences
<PaperSword>
I didn't see much in the repo and was going to make this a weekend project. My concerns are
<PaperSword>
- License
<PaperSword>
- Forking
<PaperSword>
- Is it even worth the cost?
<PaperSword>
- Performance
<PaperSword>
Yep lessons have been learned in the past from Berkley to Level .7 - .8
<sipa>
informed by the BIP50 fiasco (bdb/leveldb locking fork)
<PaperSword>
:D
<sipa>
jinx
<PaperSword>
My biggest question would be, lets say in terms of performance, LMDB is much better, as such the UTXO set that would not work with level works with LMDB that would be a hardfork?
<sipa>
why would any UTXO set not work witrh leveldb?
<PaperSword>
At somepoint the DB will just keel over right?
<PaperSword>
Similar to running out of diskspace
<sipa>
i don't see why that would be the case?
<sipa>
you mean just by virtue of becoming too slow?
<PaperSword>
Yeah
<sipa>
ok, yeah - i wouldn't call that a hard fork, more a centralization pressure thing
<PaperSword>
Okay.
<PaperSword>
Well I am going to go write something up!
<PaperSword>
Even if it's a problem would be nice to see what impact a new maintained db would have.
<PaperSword>
TY for your time.
<sipa>
yeah, i'm curious to see numbers
<PaperSword>
Just off the cuff, are you more interested in LMDB or RocksDB?
<sipa>
bitcoin core can very aggressively cache the UTXO data cache in its application-level cache though, which may or may not impact what gains are possible at all due to the database layer
<PaperSword>
Or some other backend?
<sipa>
rocksdb will definitely be easier, lmdb will likely be more interesting :p
<PaperSword>
10:01:36 AM - sipa: bitcoin core can very aggressively cache the UTXO data cache in its application-level cache though […]
<PaperSword>
You talking about the dbcache?
<sipa>
yes
<sipa>
well, the dbcache setting controls both database-layer caches and application-layer caches
<sipa>
by application-layer cache, i mean CCoinsViewCache
<PaperSword>
Yeah, the hot cache of in-memory coins
<PaperSword>
if entry is missing then you fallback to the db lookup
<sipa>
its most important benefit i think is the "fresh" optimization (i may be biased, i introduced it), where a coin that is being spent before its creation was flushed out to disk, will just be removed from memory as if it never existed, having no long-term memory usage impact, and avoiding all CPU and I/O related to writing it out
Cory99 has joined #bitcoin-core-dev
Cory38 has quit [Ping timeout: 240 seconds]
maaziEmeka has quit [Remote host closed the connection]
abubakarsadiq has quit [Quit: Connection closed for inactivity]
kevkevin has quit [Remote host closed the connection]
Christoph_ has quit [Quit: Christoph_]
Christoph_ has joined #bitcoin-core-dev
<TheCharlatan>
PaperSword afaik both rocksdb and lmdb have been extensively tested as alternatives fairly recently, both proved slower in the benchmarks.
Christoph_ has quit [Ping timeout: 252 seconds]
<sipa>
TheCharlatan: oh!
bugs_ has quit [Quit: Leaving]
<fjahr>
TheCharlatan: Has that been documented anywhere? Would be good since it seems like this topic will surely be revisited in the future.
<TheCharlatan>
yeah, not sure if something more complete has been written down.
<TheCharlatan>
I think dgumberg should know more, but he does not seem to be on this channel?
<fjahr>
I don't remember who's session this was, might be worth a delving post to document the finding for the future in a more coherent form that the transcripts.
conman has quit [Ping timeout: 276 seconds]
Cory10 has quit [Quit: Client closed]
cman has joined #bitcoin-core-dev
Cory10 has joined #bitcoin-core-dev
cman has quit [Ping timeout: 265 seconds]
cman has joined #bitcoin-core-dev
<gmaxwell>
sipa: wumpus spent a lot of effort on lmdb a couple years ago.
<gmaxwell>
rocks is basically a drops in, but yeah hasn't been faster, also really not so shocking.
<gmaxwell>
the other thing to worry about is crash durability, which is annoyingly difficult to test... not that leveldb is a paragon of it, it's at least a known quantity.
<sipa>
a point is also that the current application layer was heavily designed around leveldb's strengths and weaknesses, which means that just a straight up swapping out for a different database engine may understate its potential
conman has joined #bitcoin-core-dev
cman has quit [Ping timeout: 252 seconds]
<sipa>
e.g. rocksdb can use asynchronous i/o on some systems, if you use a MultiGet statement, to fetch multiple keys concurrently - something the current code just cannot use
<gmaxwell>
like if something is 1% more likely to get corrupted then it's probably not worth it even if it's a fair bit faster, given how costly resyncing is, and how insignificant the runtime cost for it *except* during resync.
<gmaxwell>
sipa: indeed, well also in the current code I suspect all that matters for average performance is write performance because caching is effective enough.
<sipa>
depends on the setting... high/low dbcache, ssd/hdd i/o profile, IDB vs steady state, ...
<sipa>
pruning or not
<gmaxwell>
but also average performance isnt the only metric, if you instead measure something more like 95th percentile block acceptance, it'll presumably be dominated by times when the cache was just flushed, and at that moment the database performance matters a lot more.
<gmaxwell>
there are also a bunch of tunables in leveldb / other databases that were probably last tested/adjusted with a utxo set 1/5th the current size.
conman has quit [Ping timeout: 248 seconds]
<sipa>
gmaxwell: there has been some benchmarking/tuning lately, actually, see #30039 for example
<corebot>
https://github.com/bitcoin/bitcoin/issues/30039 | dbwrapper: Bump LevelDB max file size to 32 MiB to avoid system slowdown from high disk cache flush rate by maciejsszmigiero · Pull Request #30039 · bitcoin/bitcoin · GitHub
<sipa>
also, for several types of database flushes we no longer wipe the cache