< aj> what's the status of BIP62 (malleability)? is there something documenting what's stopping it from being ready to be deployed, at least for third-party malleability?
< gmaxwell> aj: the fix for nussance third party malleability is already deployed, in 0.11.1 and 0.10.3, but most hashpower isn't running it yet.
< gmaxwell> BIP62 which also protects against miners creating malleability for a subset of transactions likely has issues still, and needs to be rewritten... fortunately the parts of it that don't have issues are flowing in.
< gmaxwell> aj: the downside is that some wallets with absentee maintance are going to get their transactions blocked; but a couple years of nagging is all we could do and with active attacks going on it didn't make sense to wait any longer and let everyone continue to get disrupted just to avoid disrupting wallets responsible for only a couple percent of transactions.
< aj> gmaxwell: makes sense... is there an existing PR for bip 62 or something i can read?
< aj> gmaxwell: (the consensus change side, more than the already-deployed isStandard changes i mean)
< gmaxwell> aj: well they're one in the same in Bitcoin Core; these changes are accomplished via validation flags. To make them non-standard we add the new restrictions to the set of flags used to verify transactions going into the mempool.
< gmaxwell> To add them to consensus, the softfork negoiates turning them on for block validation.
< aj> gmaxwell: hmm, i guess i'm confused as to what are the parts that have issues and need rewriting before it can softfork and protect from miners too then?
< gmaxwell> aj: there are more sources of malleability for transactions generally (but not ordinary p2pkh and multisig) than this addresses; at the same time, fancy contract usage -- like swaps and refunds need #9 (and perhaps #8) to be addressed, and CLTV addresses their needs better.
< gmaxwell> aj: as far as softforking it, it really need to be comprehensively non-standard first before we can do that; and softforking to prevent nussance mallability from miners is probably of pretty low value since miners have no reason to create it, so it's a lower priority especially considering how complex and invasive the changes are.
< gmaxwell> (and because every time we've picked up BIP62 again we've found more cases that weren't covered. :( )
< gmaxwell> (again, related to less common usage)
< aj> gmaxwell: oh, hmm. that sounds more complicated than i was hoping :)
< gmaxwell> aj: rethinking this resulted in coming up with the seggregated witness approach thats in elements alpha, and which may be possible to soft-fork into bitcoin.
< gmaxwell> So personally thats the route I'd like to go down.
< aj> gmaxwell: i was assuming that was a lot further off than bip62 though?
< sipa> bip62 can be simplified a lot now, if we want just that
< sipa> all of bip62's rules are already nonstandard in 0.10.3 and 0.11.1
< gmaxwell> BIP62 is infintely far off right now, no one is working on it. And I don't think the approach is likely to be very successful; except for blocking malleability in a few narrow cases (which we've already been breaking out)
< sipa> so if the network accepts those rules, we don't even need v2 transactions in bip62... just unconditionally make violations of its rules fatal
< gmaxwell> For the nussance things, non-standardness is sufficient. For contracts BIP62 is insufficient, but CLTV covers a lot of them.
< gmaxwell> (CLTV and CSV)
< aj> gmaxwell: so how far off is segregated witness? it doesn't have a bip, and needs a block size increase, i think?
< sipa> what does the block size have to do with it...?
< sipa> the complication is mostly that it needs a change in the p2p relay protocol for blocks and transactions
< aj> sipa: i thought i just read that segregated witness increased tx size by a bunch
< sipa> i thibk you're confusing with confidential transactions
< aj> sipa: aha; "In fact, this witnessing occupies 2/3rd of the blockchain." https://bitcointalk.org/index.php?topic=1210235.0
< aj> sipa: could be
< sipa> seggregated witness just moves scriptSig out of transactions
< sipa> in alpha, the seggregation is scriptSig AND the range proofs
< aj> sipa: aha, that makes more sense
< morcos> sipa: that idea i mentioned about scanning the feerate sort for "hot hashes" (txins likely to be redeemed in the next few blocks) and then not deleting those from the cache on a flush
< morcos> it has some promise, i just coded up a rough version, takes about 30ms to generate the set of txins from the top 10MB worth of txs
< morcos> its hard for me to calculate exactly how well its working b/c it totally screwed up the cache size accounting unless i get a bit smarter
< morcos> that 30ms is out of a flush time thats usually in the 300ms range, but then causes validation times to get a bit faster. anyway, i'll play around with it some more.
< gmaxwell> aj: ... uh, ... thats saying that _currently_ it's 2/3rd of the blockchain, which is all bandwidth that could be _saved_ by a synchronizing node that isn't checking historic signatures.
< gmaxwell> aj: so its the opposite, and one of the reasons that that approach is so much more attractive.
< gmaxwell> (though until a week or so ago I didn't believe it was soft-forkable, but luke appears to have more or less solved that design question)
< sipa> gmaxwell: i haven't heard luke's idea, but I think it's simple enough? use "<reedeemscripthash> OP_NOP7" as scriptPubKey, "" as scriptSig, and define an auxiliary data structure with "<scriptSig> <redeemScript>" in it
< gmaxwell> sipa: yes sir. And require the scrippubkey in the original datastructure to be empty for OP_SEGWITNESS scripts.
< gmaxwell> er original scriptsig.
< sipa> as i said :)
< gmaxwell> lol "" didn't take up enough space on the screen.
< bsm117532> Why is leveldb in the core in the first place?
< sipa> because we want to have control over its changes
< bsm117532> By not changing it...
< sipa> and we have local modifications to it (win env, disable compression, and we've had other ones before)
< sipa> if a bug were to be found in leveldb, fixing it may cause a consensus failure
< sipa> and bugs like that have happened
< sipa> (before we were using leveldb, to be clear)
< gmaxwell> bsm117532: leveldb has (prior to our use of it) fixed 'bugs' that would have broken network consensus.
< gmaxwell> oh jinx, sorry.
< bsm117532> That's yucky on so many levels.
< sipa> how do you mean?
< bsm117532> FWIW I'd have put it in a separate repo under bitcoin, rather than import the code directly.
< bsm117532> Yeah, like that ;-)
< sipa> not like that; that
< gmaxwell> It is.
< bsm117532> The github repo should have a reference to it, rather than have the code imported.
< sipa> that is how git subtree works
< sipa> subtrees are ugly
< bsm117532> checking...
< sipa> but the only alternative is submodules, which are even more ugly
< bsm117532> Yeah that's what I mean, a submodule.
< sipa> well we're using subtrees
< sipa> i don't feel like reiterating the advantages of one over the other in either direction; you can find enough discussion about that on the internet :)
< bsm117532> AFAICT, when I clone bitcoin, I don't get the bitcoin/leveldb repo. I get bitcoin/src/leveldb which is entirely separate. Am I wrong about that?
< sipa> correct, though a script is included to verify their correspondence
< bsm117532> I see. Egad that's ugly. Well then you'll just have to deal with spurious patches to leveldb. :-P
< sipa> if we need to change something to our leveldb tree, the correct way is submit it as a PR to the bitcoin/leveldb repo, and then bitcoin core can update to a new version
< bsm117532> That makes sense. It's the other direction that is generating a problem here. (modifying src/leveldb...)
< sipa> we don't ever accept changes directly to that directory
< bsm117532> Well we're going to replace leveldb with sqlite, right?!?! ;-)
< sipa> maybe.
< CodeShark> is sqlite performant enough?
< bsm117532> Also replace Berkeley db with /dev/null...
< sipa> bsm117532: already possible, use --disable-wallet at compile time :)
< sipa> CodeShark: doubtful
< bsm117532> sipa: I'm well aware, always do.
< CodeShark> we're pretty set on ditching leveldb, though, right?
< bsm117532> CodeShark: don't know. I re-ran some benchmarks from an old article: https://gist.github.com/mcelrath/6952eab246a7c705a0fb
< sipa> CodeShark: only if a suitable replacement is found
< bsm117532> I think we need a more targeted leveldb vs. sqlite (or something else) comparison.
< sipa> bsm117532: jeff has a branch with bitcoin core running on sqlite3
< sipa> but there are problems with background checkpointing etc
< CodeShark> the main issues with leveldb are that it's no longer being maintained and that it doesn't guarantee consistency, right?
< bsm117532> Well he did that quickly...
< sipa> CodeShark: it does guarantee consistency; it just seems to fail on windows pretty often
< CodeShark> doesn't it rely on the OS queueing up writes in the correct order or something?
< sipa> no
< sipa> or at least, it doesn't intend to
< sipa> it relies on the OS behaviour properly when asked to do a synchronous write/flush, though
< bsm117532> Are we worried about db corruption on power failure? Or something else generating inconsistencies?
< sipa> yes, that's what seems to happen
< CodeShark> I haven't looked at the actual study, but https://en.wikipedia.org/wiki/LevelDB#Bugs_and_Reliability
< bsm117532> That's a very hard problem that I don't think bitcoin can solve by choice of db.
< bsm117532> This is why Oracle charges big bucks for dedicated hardware.
< sipa> sqlite is well known for being very stable
< CodeShark> sqlite has worked very well for me in certain applications that don't require extremely frequent insertions
< dcousens> sipa: aye, the mere simplicity of sqlite is very attractive
< sipa> CodeShark: we have extremely infrequent insertions
< sipa> but they're very big batches
< bsm117532> I've had good luck with sqlite too. But these are anecdotes.
< tripleslash> bsm117532, its not even power failure. Windows gives very little time for apps to cleanly shutdown these days.
< sipa> lmdb seems to be a interesting candidate
< sipa> but it's mmap-based, so not usable on 32-bit systems for such large databases as we have
< dcousens> sipa: leveldb is just used for the txdb atm right?
< tripleslash> Pretty much any new pc today is shipping x64. There comes a time where it will be time to just cut the cord on the X86 systems.
< sipa> dcousens: block index and chainstate
< sipa> tripleslash: yet ARM is becoming more and more relevant
< CodeShark> can't we create multiple databases all restricted to 4 gigs? :)
< tripleslash> sipa: true that
< sipa> CodeShark: we need atomic changes across databases then
< bsm117532> I vote that a 64 bit system as a requirement is a reasonable thing. Bending over backwards to shoe-horn into 32 bits is not worth anyone's time.
< dcousens> sipa: right, the CBlockTree is defined in txdb.h haha
< CodeShark> we can always support mmap on 64-bit systems and fall back on leveldb for 32-bit systems
< CodeShark> although dunno what that might entail regarding consensus
< CodeShark> it only takes one "undocumented feature" to screw everything up
< sipa> having multiple databases is technically not hard; the database interface is pretty neatly abstracted
< sipa> but it's very unattractive to risk divergence between them, especially when testing mostly happens on one, and production mostly on another :)
< dcousens> sipa: production on 32-bit?
< bsm117532> Quick git question...I did a push -f to overwrite my previous commit, because I like having clean histories. I could have also made a second commit on this PR. Does anyone care? Would github have squashed them anyway? (Re: https://github.com/bitcoin/bitcoin/pull/6899 )
< sipa> bsm117532: github won't squash for you
< sipa> reviewers may ask to squash things
< * bsm117532> has spent too long being the only committer to his repo.
< bsm117532> Ok so I did the right thing.
< sipa> only reason to not squash is if it's a complicated commit that required lots of review (for example, big code movement commits)
< dcousens> sipa: is the squash op not deterministic?
< sipa> dcousens: the resulting tree of a squashed commit is identical to the resulting tree of the series of commits it derived from
< bsm117532> There is no squash op... rebase -i is very manual and not deterministic at all.
< dcousens> I know the rebase is, and hence verifiable
< sipa> dcousens: you're confusing commits with trees
< CodeShark> rebase is only not deterministic when you either change commit order or rebase off a different branch
< sipa> or if there were conflicts
< sipa> or merges within the commits you're rebasing
< sipa> rebase applies a merge resolution algorithm, and you can change that algorithm
< CodeShark> commit followed by squash is effectively the same as commit --amend
< sipa> which is also not deterministic :)
< sipa> (the resulting tree is, but the commit itself isn't)
< dcousens> sipa: interesting
< sipa> the reason it isn't is because a commit has a timestamp
< CodeShark> but that's a trivial source of nondeterminism ;)
< sipa> you can avoid it by specifying the timestamp on the command-line, though
< sipa> yeah
< sipa> but rebases are generally not verifiable
< sipa> and afaik nobody does that
< bsm117532> LMDB does look really interesting...
< bsm117532> Looks like it uses COW instead of sqlite's *.journal which I think is write-ahead logging. So should be much faster than sqlite, probably a factor of 2.
< sipa> indeed
< bsm117532> It looks like an in-memory btrfs ;-)
< sipa> except it's not in-memory
< bsm117532> Yeah but mmapped, so it looks like it's in-memory.
< bsm117532> I've got one more tedious bug I want to fix, but maybe then I'll look at making an LMDB branch for comparison. If it was so easy for jeff to rip out leveldb for sqlite...
< jgarzik> COW is COW, somewhat like write-ahead logging. :)
< jgarzik> If you order your writes properly, it's write-once
< sipa> ... i do wish we could just use LMDB
< jgarzik> It's quite easy to switch out. I could do an LMDB patch. Also working on a COW dbm myself.
< sipa> i'm sure it wouldn't be hard; but LMDB is a total no go on 32-bit systems
< jgarzik> the best performance should be COW + Kyoto Cabinet scheme
< jgarzik> (KC is worth a look too)
< jgarzik> COW practically guarantees no corruption (assuming write order is proper)
< sipa> but i would be interested in LMDB's performance...
< jgarzik> no idea if LMDB performs better than KC. Worth benching, definitely.
< jgarzik> Google benching leveldb, sqlite, kyoto cabinet: http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html
< phantomcircuit> jgarzik, COW without any delete? :P
< jgarzik> phantomcircuit, typical strategy is reclaim after X generations of the superblock
< jgarzik> assuming no stored snapshots
< jgarzik> Useful link. They should have benched the KC hash db and not btree. ;p
< sipa> not a fair comparison, as lmdb/leveldb only do ordered maps
< gmaxwell> has anyone managed to complete a sync with jeff's sqllite attempt?
< gmaxwell> I think jcorgan said it was going on day 2 for him.
< tripleslash> if you build a win64 binary for me, I'll give it a go.
< jgarzik> bench'd BDB in b-tree mode too :(
< jgarzik> if we didn't need ordering, things could go really fast
< sipa> we don't really need ordering
< sipa> though it may end up being useful if we go from per-tx caching to per-txout caching
< sipa> and for normative utxo hashes
< phantomcircuit> jgarzik, more importantly we can significantly over provision the initial db to avoid resizing constantly
< jgarzik> LMDB could use a windowed mmap approach and support 32-bit systems, >2GB databases
< jgarzik> or we could just drop 32-bit support
< jgarzik> "full node = big database = big iron = no rPi"
< Luke-Jr> jgarzik: …
< Luke-Jr> I use 32-bit.
< tripleslash> Luke-Jr, you also use dialup. At some point, people have to upgrade. ;-)
< Luke-Jr> except for Valgrind not supporting it, it's a more logical choice than 64-bit. especially x32, once more stuff works.
< Luke-Jr> tripleslash: I do not use dialup.
< tripleslash> my apologies.
< jgarzik> for large databases the address space helps a -lot-. x32 useful but not in this case.
< Luke-Jr> tripleslash: also, I upgraded to 64-bit within a week of AMD releasing the first 64-bit capable CPU. I decided a year or two ago that CPUs were fast enough that 32-bit was the better option now.
< Luke-Jr> jgarzik: x32 is only useful if basically everything is x32; not so much if you have both x32 and amd64 programs
< jgarzik> not true (but off-topic so I'll stop)
< sipa> note: LMDB's on-disk format is platform dependent
< sipa> (byte order and word size)
< phantomcircuit> jgarzik, unfortunately lmdb really isn't an option because of the trade offs they made
< sipa> phantomcircuit: any specifics, apart from intentionally no support for 32 bit systems?
< dcousens> Luke-Jr: "CPUs were fast enough that 32-bit was the better option
< gmaxwell> sipa: well no integrity checks iirc, and non-portable data (latter less of an issue); I think there was some other thing wumpus raised.
< dcousens> If its not relevant, could you PM what you mean by that, ooi
< Luke-Jr> dcousens: #bitcoin maybe?
< dcousens> Luke-Jr: sure
< gmaxwell> sipa: or we could just admit that we'll need a custom data structure to make any kind of commitment space/time efficient. :-/
< wumpus> -1 for dropping 32 bit support, I want to support 32-bit ARM
< wumpus> leveldb works well enough I see no need for any kind of desperate measures
< wumpus> experimenting with other databases, fine, but I don't like this talk of dropping support for platforms just to accomodate a library
< CodeShark> I agree with not removing leveldb support - but other than the fact we have close to zero tolerance for differences in consensus behavior, I think supporting multiple databases is generally a good idea
< wumpus> we've made it very easy to switch the database, but not looking forward to maintaining and testing multiple db backends in the upstream repository
< gmaxwell> it's very easy to use another one, though most alternatives have performance so poor they're just unusable.
< wumpus> I can't say this enough, the database is not an external interface, it's just an implementation detail of bitcoind. There should be zero reason for users to switch it.
< CodeShark> apparently corruption is not that infrequent on windows
< wumpus> then fix it on windows! it's not magic or rocket science
< wumpus> I'm tired of this complaining
< CodeShark> ?
< gmaxwell> CodeShark: sure, so go fix it! it's likely the same issue that existed on OSX or similar.
< wumpus> can we pool a bounty or something like that
< CodeShark> you mean fix leveldb itself?
< gmaxwell> (where fsync didn't work for writes via mmaps)
< wumpus> I don't want to hear "leveldb is broken on windows" every day, this is not tea time with bitcoin-dev, we should be constructive here
< gmaxwell> CodeShark: leveldb's FS access is abstracted, the windows interface we use is probably only used by us and nothing else.
< wumpus> yes
< gmaxwell> In chrome, for example, all the FS access is via some totally chrome specific layer and doesn't touch any of the FS access code we use.
< CodeShark> I don't really fricking care, personally - I don't really run a bunch of bitcoin core instances on windows - I'm just repeating what I've heard from others
< gmaxwell> CodeShark: you actually run windows though, none of the other people who would normally fix something like this do.. and we don't even have an archive of a reproduction.
< Luke-Jr> (tomorrow we find out CodeShark stopped running Windows :p)
< wumpus> I do care but I don't have access to a remotely recent windows machine at the moment, and know very little about windows internals or debugging. I generally use wine when I need to do that but it's no help here.
< CodeShark> my windows development at present consists almost 100% of crossbuilds from linux
< CodeShark> windows is just the way I access VMs and servers ;)
< CodeShark> from one machine
< CodeShark> none of my other machines run windows
< GitHub14> [bitcoin] laanwj opened pull request #6900: gitian: build on ubuntu 14.04 (master...2015_10_gitian_trusty) https://github.com/bitcoin/bitcoin/pull/6900
< wumpus> an easy to implement robustness option would be to auto-back up the utxo database once in a while. This can be done in a similar fashion to gettxoutsetinfo, on a snapshot inthe background. Then if unrecoverable database issues happen, restore that
< wumpus> (not meant as a substitute for solving the leveldb issue on windows, but what I like is that it helps for both known and unknown issues)
< zander> I'm wondering about the process of removing stuff from the mempool when a new block comes in. So transactions from the block naturally are remove from the mempool. Is that mallaeble safe?
< zander> Is the txid used for that removal?
< zander> Or is it smarter that that.
< zander> ?
< phantomcircuit> zander, any transactions which conflicts with the new block is removed
< zander> ah, ok. Thanks!
< max777555> Friends start a new cloud of mining https://cldmine.com/account/registration/4075 Very similar to when registering RDPmain dogikoin 1500 bonus on you buying power and immediately start earning a great opportunity to earn at the very beginning to invite many partners and earn a passive join !!!
< phantomcircuit> wumpus, sipa ^
< GitHub195> [bitcoin] laanwj closed pull request #6781: [QT] pretty print (indent) multiline html output (master...2015/10/qt_rpcconsole_pp) https://github.com/bitcoin/bitcoin/pull/6781
< GitHub111> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/8daffe227bc6...26752767df45
< GitHub111> bitcoin/master 3e187f2 Suhas Daftuar: Fix BIP65 p2p test...
< GitHub111> bitcoin/master 2675276 Wladimir J. van der Laan: Merge pull request #6894...
< GitHub96> [bitcoin] laanwj closed pull request #6894: [Tests] Fix BIP65 p2p test (master...fix-bip65-p2p-test) https://github.com/bitcoin/bitcoin/pull/6894
< bsm117532> We need a test case on windows that generates leveldb corruption, so as to evaluate other db alternatives. Anecdotes that it happens with leveldb and might not with some other db are unsatisfactory. What can we do to "pull the rug out from under bitcoind" and test this?
< bsm117532> Questions about performance are a lot easier to be quantitative about...
< wumpus> reproduction should be simple: run it on a windows pc, crashhe computer or disconnect the power
< wumpus> it appears that no one remotely capapble and willing to troubleshoot this issue has no windows PC to check though
< bsm117532> That's pretty tedious. :-/
< bsm117532> I don't have a windows pc either...
< wumpus> well from what I understand it *always* happens, so it's not that bad
< bsm117532> I was thinking like kill -9 on a remote windows vm, or something like that.
< wumpus> well a VM could work, I don't know - couldn't reproduce it with wine at least last time I tried
< wumpus> but a full VM is closer to the real thing. I don't have any windows licenses for a VM either, though.
< bsm117532> We should be able to get a windows VM through Azure
< GitHub178> [bitcoin] laanwj pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/26752767df45...b2ce2c1f0fb1
< GitHub178> bitcoin/master 95f4291 MarcoFalke: [trivial] Rewrite help text for feature enabled by default...
< GitHub178> bitcoin/master bf68191 MarcoFalke: [trivial] rpcnet: fix typo
< GitHub178> bitcoin/master 6782f58 MarcoFalke: [trivial] Latest config.guess...
< GitHub152> [bitcoin] laanwj closed pull request #6870: [trivial] Misc cleanup and translations (master...MarcoFalke-2015-trivial3) https://github.com/bitcoin/bitcoin/pull/6870
< GitHub57> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/b2ce2c1f0fb1...b28c229324d0
< GitHub57> bitcoin/master a83f3c2 Bob McElrath: Add explicit shared_ptr constructor due to C++11 error
< GitHub57> bitcoin/master b28c229 Wladimir J. van der Laan: Merge pull request #6899...
< GitHub144> [bitcoin] laanwj closed pull request #6899: Add explicit shared_ptr constructor due to C++11 error (master...cpp11) https://github.com/bitcoin/bitcoin/pull/6899
< GitHub189> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/b28c229324d0...725539ea0376
< GitHub189> bitcoin/master 0be387a Daniel Kraft: unittest: fix test for null tx input...
< GitHub189> bitcoin/master 725539e Wladimir J. van der Laan: Merge pull request #6863...
< GitHub146> [bitcoin] laanwj closed pull request #6863: [Test Suite] Fix test for null tx input (master...null-txin-test) https://github.com/bitcoin/bitcoin/pull/6863
< dcousens> wumpus: for the windows box, would I need a complete chain?
< dcousens> I can put bitcoin-qt on a old gaming rig if necessary, have to dust it off a little but, if its worth it
< bsm117532> It would be good to know quantitatively if we're actually fixing something by changing the db or just making a mess.
< dcousens> mcelrath: would I need to build latest?
< dcousens> I'd assume so if the aim was to actually fix it haha
< mcelrath> build latest bitcoind? Yes, I mean we'd want to take jgarzik's sqlite branch and a lmdb branch and violently kill them and see what happens to their db's too.
< mcelrath> It's kind of an involved test :-/
< dcousens> mcelrath: well, I'll see what I can set up tomorrow. Midnight here so I won't go about messing with setting up a dev-env
< mcelrath> But writing a shell script to kill things in a tight loop sounds like an appropriate thing to do to Windows. ;-)
< mcelrath> Cool, thanks!
< mcelrath> I'm pretty sure we (or lots of people) can get an azure instance. But I've never used a remote windows vm.
< wumpus> dcousens: I don't think so, I think you just have to kill it while it's syncing
< dcousens> wumpus: likely that the shutdown time itself will be dependent on how many writeops are waiting
< dcousens> Perhaps worth doing on a HDD, not a SSD?
< dcousens> :P
< mcelrath> Is bitcoin anymore consensus-dependent on leveldb? If we swap it out with sqlite or lmdb, will it still be consensus critical? (and can it be made non-consensus critical?)
< wumpus> everything that is called by the consensus code is, in principle, consensus critical
< wumpus> so is the database, the file system, the OS. But all that should be important is that they behave correctly.
< wumpus> if there are silent bugs it is problematic. For example, if leveldb would ignoring keys with certain properties, using a database without that erratic behavior would fork off from nodes using leveldb
< mcelrath> IOW if the verification code (erroneously) can't look up a txid or utxo, it causes a block to be rejected, no?
< wumpus> not if it is reported as a database error and simply crashes the node. It is a problem if a record is reported to be present when it is not, or the other way around, or the data is corrupted
< wumpus> (which is why leveldb's checksums on everything are nice to have, it provides added assurance that if something is returned it's at least correct)
< mcelrath> I'm pretty seriously worried about silent bit flips or hash computation errors causing nodes to fail, and how to detect it. But spurious errors like that won't cause a majority of nodes to reject blocks.
< wumpus> majority plays no role in bitcoin - if *your* node forks from the block chain, that's a risk to you
< mcelrath> Exactly. But having a majority of miners screw up is a bigger problem than little 'ol me. ;-)
< wumpus> but yes the most dangerous bugs are consistent ones - note that an database-OS bug could be dangerous in that way, for example forking off all MacOSX nodes
< GitHub33> [bitcoin] dajohi opened pull request #6902: policy: Add new constant MAX_STANDARD_MULTISIG_KEYS (master...multisig_keys) https://github.com/bitcoin/bitcoin/pull/6902
< morcos> sipa: what does the 25-50% of the cache number that's used for nCoinDBCache represent?
< mcelrath> Hmm I can't assign this to myself: https://github.com/bitcoin/bitcoin/issues/6903
< sipa> morcos: leveldb cache
< morcos> sipa: you mean its internal caching?
< sipa> yes
< morcos> oh i see, ha, i thought that local variable wasn't used anywhere, i didn't think to look in init
< GitHub195> [bitcoin] LordOfTheCoin opened pull request #6904: LevelDB Fixes and Enhancements (master...2015_sqlite) https://github.com/bitcoin/bitcoin/pull/6904
< mcelrath> Why is someone other than jgarzik making PR's from his repo?
< sipa> heh, i didn't even realize that by seeing github's email
< mcelrath> FYI, replacing auto_ptr -> unique_ptr (C++11) in the 5 places it appears does not introduce any new compiler errors or warnings due to the slightly different assignment semantics. So making this replacement should be fine to switch to C++11.
< GitHub101> [bitcoin] LordOfTheCoin closed pull request #6904: LevelDB Fixes and Enhancements (master...2015_sqlite) https://github.com/bitcoin/bitcoin/pull/6904
< GitHub8> [bitcoin] LordOfTheCoin reopened pull request #6904: LevelDB Fixes and Enhancements (master...2015_sqlite) https://github.com/bitcoin/bitcoin/pull/6904
< GitHub5> [bitcoin] LordOfTheCoin closed pull request #6904: LevelDB Fixes and Enhancements (master...2015_sqlite) https://github.com/bitcoin/bitcoin/pull/6904
< morcos> sipa: ok this might be a bit obscure, so i'll just look into if you have no idea off the top off your head. but if I have a bunch of entries cached in a pcoinsTip and then I call TestBlockValidity on a block that accesses only those entries
< morcos> TBV will build a CCoinsViewCache on top of pcoinsTip, which cache will be passed to ConnectBlock which will build yet another on top. modify the top most cache, and flush to the intermediate cache, then return to TBV which will just dump the intermediate cache on the ground
< morcos> any reason that would be FASTER if some of the entries in the pcoinsTip cache had FRESH or DIRTY flags as opposed to if they all had no flags.
< sipa> so pcoinsTip > TBV-Cache > ConnectBlock-Cache
< sipa> and it's in pcoinsTip that pre-existing flags would matter?
< MarcoFalke> cfields, would you mind if I rebase trivial-next?
< cfields> MarcoFalke: sorry i missed your pm the other day. you could, but it's kinda a pain. i'd prefer to take care of it this week. I think the separate repo turned out to not make things any easier.
< MarcoFalke> Ok, then. I will leave it to you to take care of it.
< GitHub129> [bitcoin] MarcoFalke opened pull request #6905: Use constants and minor fixes by luke-jr (master...lukejr-constants-no-mergeConf) https://github.com/bitcoin/bitcoin/pull/6905
< GitHub81> [bitcoin] gmaxwell opened pull request #6906: Reject invalid pubkeys when reading ckey items from the wallet. (master...ckey_pubkey_check) https://github.com/bitcoin/bitcoin/pull/6906
< morcos> sipa: yeah thats what i meant, but maybe thats not the issue. i'll dig into it. but i really like this idea of keeping the hot items in the cache. then you can flush it much more regularly and don't have to worry about it growing too big.
< GitHub171> [bitcoin] MarcoFalke closed pull request #6905: Use constants and minor fixes by luke-jr (master...lukejr-constants-no-mergeConf) https://github.com/bitcoin/bitcoin/pull/6905
< morcos> sipa: oh i lied about that anywya, there is no extra cache in ConnectBlock. it just uses the TBV cache, or in the normal case the cache on pcoinsTip is added in ConnectTip before ConnectBlock
< MarcoFalke> Any thoughts on the clang-format thing?
< mcelrath> FYI: https://github.com/andrewseidl/githook-clang-format "Warning: Do not use this on an existing codebase that isn't already in your desired style. Doing so will lead to a string a dirty commits where your code changes are intermixed with clang-format's formatting changes. Furthermore, every developer will need to install this hook. If they don't, you will again end up with commits with a mixture of code and formatting changes."
< jgarzik> already covered in #bitcoin-dev meeting
< MarcoFalke> Do those channels have different scopes? I remember bitcoin-dev is deprecated?
< wumpus> this channel has a very narrow scope: code changes to the bitcoin core project
< sipa> clang format discussion would be very ontopic here IMHO
< sipa> but it was somehow part of the wider bitcoin irc meeting, which took place in #bitcoin-dev an hour ago
< davec> Is it intentional that the "minRelayTxFee" is not actually a minimum? I suspect the answer is yes and the variable simply wasn't renamed.
< MarcoFalke> It's a minimum per node basis to get into that nodes mempool.
< davec> so if you have a tx of say 250 bytes, the will result in 250 * 1000 (the default "minRelayTxFee") / 1000 = 250 which is clearly < 1000
< MarcoFalke> yes, it's per kB
< wumpus> right it's the minimum *per KB*, all fees in bitcoin core are per kB
< wumpus> all fee settings at least
< MarcoFalke> wumpus, not paytxfee ;)
< MarcoFalke> not yet at least
< davec> alright thanks for confirming. The naming and description made it seem like it was an absolute minumum, so I wanted to confirm.
< davec> because that's obviously not what the code is doing
< wumpus> MarcoFalke: hmm
< MarcoFalke> Does src/qt require a different .clang-format style file?
< GitHub71> [bitcoin] jtimon opened pull request #6907: Chainparams: Use a regular factory for creating chainparams (master...chainparams-factory-0.12.99) https://github.com/bitcoin/bitcoin/pull/6907
< GitHub107> [bitcoin] jtimon opened pull request #6908: Chainparams: DRY: Make qt/guiutil.cpp fit BIP70 chain name strings (master...chainparams-bip70-0.12.99) https://github.com/bitcoin/bitcoin/pull/6908
< sipa> heh, a rebase with master jumps to 300-600% CPU usage immediately
< sipa> *reindex
< gmaxwell> thats because reindex checks all the signatures.
< sipa> not historical ones?
< gmaxwell> yes, historical ones.
< sipa> oh, because we don't have the header yet for the chainpoint it is an ancestor of
< GitHub7> [bitcoin] MarcoFalke reopened pull request #6905: Use constants and minor fixes by luke-jr (master...lukejr-constants-no-mergeConf) https://github.com/bitcoin/bitcoin/pull/6905
< phantomcircuit> sipa, :)
< gmaxwell> Can someone build me windows binaries for https://github.com/bitcoin/bitcoin/pull/6906 ? (oh I miss the testers that produced binaries as a side effect...)