< bitcoin-git> [bitcoin] sipa pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/61fb80660f73...033c78671b91
< bitcoin-git> bitcoin/master bd9c181 John Newbery: [rpc] Add initialblockdownload to getblockchaininfo
< bitcoin-git> bitcoin/master 1141364 John Newbery: [trivial] (whitespace only) fix getblockchaininfo alignment
< bitcoin-git> bitcoin/master 033c786 Pieter Wuille: Merge #11258: [rpc] Add initialblockdownload to getblockchaininfo...
< bitcoin-git> [bitcoin] sipa closed pull request #11258: [rpc] Add initialblockdownload to getblockchaininfo (master...expose_ibd) https://github.com/bitcoin/bitcoin/pull/11258
< ZSky> Hi
< ZSky> When a block is added, is it written/serialized on disk immediately?
< ZSky> If so, using what kind of storing? DB-like? flat files?
< sipa> flat files
< sipa> after processing (which may happen at the same time as it's received, or later), its effects are applied to the UTXO set
< ZSky> sipa: immediately after a block is added, or does it stay in RAM for a while before flushing to disk?
< sipa> it's written immediately if it's acceptable
< sipa> though we reuse the in-memory version IF processing happens at the same time
< ZSky> sipa: what kind of flat file structure? just "fopen append" to a single file?
< sipa> we keep track of how many bytes are already used in each file in a database
< sipa> and then just write the exact range
< ZSky> you mean the offset?
< sipa> yes
< ZSky> so the raw data is in one single big file (100+GB)
< sipa> no, it's in many files
< ZSky> and there's a db for offsets
< sipa> blk?????.dat
< sipa> 128 MiB each
< sipa> and indeed, there is a LevelDB database with metadata about each block (which includes the file and offset)
< ZSky> each time a 128 MiB is full, a new one is created, and the DB stores [block13672: { filenumber = 27, offsetinfile = 283792 }, ...]
< sipa> exactly
< ZSky> oh that's nice
< ZSky> wow leveldb seems nice
< ZSky> it's been a long time I was looking for a serialized key-value storage
< ZSky> usable in production
< sipa> it's not perfect
< sipa> but better than anything else we've tried
< ZSky> so you're using this currently: https://github.com/google/leveldb
< sipa> we have our own fork with minor changes: https://github.com/bitcoin-core/leveldb
< sipa> in particular, MinGW windows support
< ZSky> sipa: dumb question: why not using leveldb this way: block13672: [BLOCKRAWDATA]
< ZSky> and totally rely on leveldb's data management?
< sipa> because we don't need leveldb's data management
< ZSky> you use it for metadata
< ZSky> so you kind of need a small key:value db
< sipa> yes, which is a tiny amount of data
< ZSky> true
< ZSky> so the idea is: you keep a key:value store for the smallest amount of data possible
< sipa> but for blocks themselves, we don't need consistency, or the ability to selectively update and delete, or ...
< ZSky> true
< sipa> it's just a stream of data that comes in, and we can drop it in effectively append only files on disk
< sipa> in fact, we hardly use the block data after it's been processed
< sipa> it's just for when another peer in the network asks for it
< sipa> or to rescan for old wallet activity
< ZSky> yes i see
< ZSky> historically, in the earliest implementations, what was used before leveldb?
< sipa> BDB
< sipa> changed in 0.8
< sipa> and the wallet still uses BDB
< ZSky> why no BDB anymore?
< sipa> many many reports of corruption
< sipa> incompatibilities between versions (you can't downgrade, and in some cases you can't even upgrade)
< sipa> and the fact that BDB was indirectly the cause for one of the largest operational problems the system has seen in years (read BIP50)
< ZSky> oh really?
< sipa> which was related due to it know how many locks the database will require during its operation
< ZSky> tl;dr what was it?
< sipa> while we don't need any locks at all
< sipa> you need to configure BDB with "i need up to X locks", and if any atomic database operation touches more internal records than you have configured locks, the operations fails
< sipa> this is because BDB is a multi-application database system (multiple processes can simultaneously open and modify the database, and there is consistency across them), which is something we don't care about at all
< ZSky> sipa: what was used in 0.1.0?
< sipa> BDB
< ZSky> ok right
< ZSky> sipa: do you do python sometimes?
< sipa> hardly
< ZSky> i was looking for such a keystore in py
< ZSky> sipa: thanks for these informations!
< sipa> yw
< cluelessperson> aj: thank you for your help
< cluelessperson> How can I help you all?
< cluelessperson> I can do some devops
< cluelessperson> put me to use
< meshcollider> devops...199
< mryandao> meshcollider: lol, funny
< meshcollider> CScripts in the wallet don't have birth-times associated with them do they
< bitcoin-git> [bitcoin] luke-jr opened pull request #11658: During IBD, when doing pruning, prune 10% extra to avoid pruning again soon after (master...ibd_prune_extra) https://github.com/bitcoin/bitcoin/pull/11658
< bitcoin-git> [bitcoin] luke-jr opened pull request #11660: RPC: Internal named params (master...internal_named_params) https://github.com/bitcoin/bitcoin/pull/11660
< luke-jr> ^ hope that satisfies concerns with #11441
< gribble> https://github.com/bitcoin/bitcoin/issues/11441 | rpc/server: Support for specifying options as named parameters by luke-jr · Pull Request #11441 · bitcoin/bitcoin · GitHub
< meshcollider> MarcoFalke: I've seen a few PRs being closed recently as 'up for grabs', can we have a label for it to keep track of them?
< meshcollider> Maybe called "Abandoned" or something
< bitcoin-git> [bitcoin] laanwj pushed 1 new commit to master: https://github.com/bitcoin/bitcoin/commit/6de3203cdce2d8532f39f9f9428c33b0dd53f623
< bitcoin-git> bitcoin/master 6de3203 Wladimir J. van der Laan: doc: Add historical release notes for 0.15.1...
< F0lks> Hi, is there some devs' available for a little talk ? It won't be long I promise
< bitcoin-git> [bitcoin] luke-jr opened pull request #11662: [0.15] Sanity-check script sizes in bitcoin-tx (0.15...bitcoin-tx-script-sizes-0.14) https://github.com/bitcoin/bitcoin/pull/11662
< wumpus> release date 2017-11-11
< wumpus> nice
< wxss> \o/
< paveljanik> remaining days to Xmas: *42*.
< LumberCartel> paveljanik++
< wumpus> :D
< MarcoFalke> meshcollider: Sure will do that
< LowKey> hi, anyone know how to build mac wallet for bitcoin?
< mlz> LowKey, ask in #bitcoin channel
< LowKey> ok
< bitcoin-git> [bitcoin] MarcoFalke pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/6de3203cdce2...95e14dc81dd3
< bitcoin-git> bitcoin/master ea0cd24 John Newbery: [tests] Tidy up receivedby.py...
< bitcoin-git> bitcoin/master 5e0ba8f John Newbery: [wallet] getreceivedbyaddress should return error if address is not mine
< bitcoin-git> bitcoin/master 95e14dc MarcoFalke: Merge #11055: [wallet] [rpc] getreceivedbyaddress should return error if called with address not owned by the wallet...
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #11055: [wallet] [rpc] getreceivedbyaddress should return error if called with address not owned by the wallet (master...getreceivedbyaddress_error) https://github.com/bitcoin/bitcoin/pull/11055
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #11663: [trivial] doc: Add getreceivedbyaddress release notes (master...Mf1711-docReleaseNotes16) https://github.com/bitcoin/bitcoin/pull/11663
< sipa> MarcoFalke: your PGP signatures scare me!
< MarcoFalke> why is that?
< sipa> any time i receive a bitcoin related email with PGP headers i assume there is some terrible vulnerability :)
< MarcoFalke> heh. Makes sense now that you say it. I will try to figure out something else.
< sipa> oh please no, it was not a complaint :)
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/95e14dc81dd3...13e352dc53de
< bitcoin-git> bitcoin/master 927f4ff Luke Dashjr: GUI: Receive: Remove option to reuse a previous address...
< bitcoin-git> bitcoin/master 13e352d MarcoFalke: Merge #3716: GUI: Receive: Remove option to reuse a previous address...
< luke-jr> jnewbery: promag: am I missing something? :/ http://luke.dashjr.org/tmp/screenshots/Screenshot_20171111_192644.png
< promag> luke-jr: what you mean? the test verifies that initialblockdownload is a key of getblockchaininfo response
< promag> or am I wrong?
< bitcoin-git> [bitcoin] benma closed pull request #9897: AppInitMain: split initialization of Connman into a new function (master...connman) https://github.com/bitcoin/bitcoin/pull/9897
< luke-jr> promag: for some reason, GitHub isn't showing me that part :|
< luke-jr> oh well
< jtimon> jl2012: not sure if you still want to do #11398 on top of #11426 since some people don't seem to agree it makes things easier, but rebased
< gribble> https://github.com/bitcoin/bitcoin/issues/11398 | Hardcode CSV and SEGWIT deployment by jl2012 · Pull Request #11398 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/11426 | BIP90: Make buried deployments slightly more easily extensible by jtimon · Pull Request #11426 · bitcoin/bitcoin · GitHub
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/13e352dc53de...2adbddb03840
< bitcoin-git> bitcoin/master 1e65f0f practicalswift: Use compile-time constants instead of unnamed enumerations (remove "enum hack")
< bitcoin-git> bitcoin/master 2adbddb MarcoFalke: Merge #10749: Use compile-time constants instead of unnamed enumerations (remove "enum hack")...
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #10749: Use compile-time constants instead of unnamed enumerations (remove "enum hack") (master...enum-hack) https://github.com/bitcoin/bitcoin/pull/10749