< wumpus>
yes that reminds me that gitian doesn't do any merging/rebasing like the travis build does, so having #6917 in master wouldn't actually help by default
< GitHub47>
bitcoin/master e4e5334 Gregory Maxwell: Restore MedianTimePast for locktime....
< GitHub47>
bitcoin/master d1c3762 Gregory Maxwell: Revert "Revert "Enable policy enforcing GetMedianTimePast as the end point of lock-time constraints""...
< GitHub47>
bitcoin/master 3038eb6 Wladimir J. van der Laan: Merge pull request #6934...
< GitHub41>
bitcoin/0.11 a1d3c6f Mark Friedenbach: Add rules--presently disabled--for using GetMedianTimePast as endpoint for lock-time calculations...
< GitHub41>
bitcoin/0.11 f720c5f Mark Friedenbach: Enable policy enforcing GetMedianTimePast as the end point of lock-time constraints...
< GitHub41>
bitcoin/0.11 df616ae Wladimir J. van der Laan: Merge pull request #6884...
< jtimon>
gmaxwell: re transaction cost formula: when you use it only as policy, it has nothing to do with the blocksize limit, when you use it to calculate the size of blocks it does (that's whatI meant by "we can make it consensus")
< fanquake>
Has anyone been testing core with Boost 1.59.0 ?
< jonasschnelli>
wumpus: still facing "Corruption: error in middle of record" with #6948
< wumpus>
fuck fuck fuck
< wumpus>
fanquake: yes, works fine here
< * jonasschnelli>
is still on boost 1.58
< jonasschnelli>
wumpus: also did shut down / power off the host (mac) and there where no issues with the database.
< wumpus>
well I never managed to get an error in the middle of record with #6948, so at least it should be more rare now
< jonasschnelli>
restarted win8.1 ... restarted bitcoin-qt from 6948
< jonasschnelli>
-> resulted in "Corruption: error in middle of record"
< wumpus>
but if that doesn't fix it I don't know, I really don't know anymore
< jonasschnelli>
switch away from leveldb
< wumpus>
maybe FlushFileBuffers on windows doesn't really flush the buffers
< wumpus>
no, I don't want to hear that
< jonasschnelli>
I'll do the test again ... maybe i missclicked somewhere.
< wumpus>
well, unless, have you tried this with another database at all?
< jonasschnelli>
sqlite3 was to slow i heard... i'd though about trying it with the unstable sqlite4 branch
< wumpus>
it would be somewhat interesting to try this with jgarzik's sqlite patch. At least to have a measurement point.
< jonasschnelli>
jgarzik uses seqlite3 which is certenly much slower then leveldb... sqlite4 sounds promising... and unstable.
< wumpus>
for this experiment I don't care about slowness - I wonder if *anything * can handle this kind of corruption succesfully
< jonasschnelli>
when i restore my snapshot based on a ~synced chain with #6917, bitcoin-core crashed,... but can be started again with 100% success. Can that be a problem? I assume no?
< fanquake>
jonasschnelli Do you have VM snapshots available for download? I can contribute some more time to testing this, having a VM to boostrap from would be handy.
< jonasschnelli>
fanquake: I have plenty of VMs (multiple harddrives full of vms), just tell me what you need. I can even get some of you access to VMs if someone needs to track down an issue.
< jonasschnelli>
Maybe you'd like to start with the Win8.1 VM... you need VMWare Fusion 8 (or higher)
< fanquake>
I'd take copies of the ones your using to test #6948
< wumpus>
I suppose what leveldb should do when it encounters corruption in the middle of a record (probably a run of 0000000 like debug.log) is to simply discard that batch and anything after it
< fanquake>
I'm actually setup somewhere with a decent internet connection atm, so downloading won't be so hard.
< fanquake>
Can't run in virtualbox?
< jonasschnelli>
nope.
< fanquake>
Ah ok, I'll spin up something on VB then.
< wumpus>
be careful to actually test #6948 on top of master, it's currently rebased to a pre-#6917 branch so as-is it won't test the robust version
< jonasschnelli>
I have also though about your "pgdb2"-like-approach, ... but it seems to be a hard way.
< jgarzik>
"The default built-in storage engine is a log-structured merge database. It is very fast, faster than LevelDB, supports nested transactions, and stores all content in a single disk file. "
< * jgarzik>
knows what log-structured is. What is a merge database, I wonder
< jonasschnelli>
jgarzik: would it be complicated to modificate your sqlite3 branch to use sqlite4? (over a subtree or something similar because it's unstable and not available over dep.management?)
< jgarzik>
jonasschnelli, Part of pgdb2 is building an underlying framework that can be used for ordered-map key/value, hash tree key/value, merkle db and more - a bit of a primitive kernel filesystem that provides transactions/COW to the upper layer, which defines the key/value structure. It's the same speed (or faster, really)
< jgarzik>
jonasschnelli, sounds like using sqlite4 is simple - take my branch and add 'env' argument to each function call.
< jonasschnelli>
jgarzik: do you have plans to work towards sqlite4 and report some benchmark numbers? Or should i give it a try?
< jonasschnelli>
but i really like to find a solution to fix the current leveldb corruption issues...
< jonasschnelli>
although, i don't care about windows enough,.... but i care about users complaining about the issue and stopping full nodes.
< jgarzik>
jonasschnelli, feel free to give sqlite4 a try. I consider my sqlite branch at a [temporary?] pause / end state. I task switched over to BIP 1xx and pgdb2.
< jonasschnelli>
Okay. pgdb2 looks nice btw!
< sipa>
jgarzik: merge database... there are several files that can refer to the same range of keys, but if there are too many you merge them into a bigger file
< jgarzik>
jonasschnelli, go ahead and try with sqlite4
< jgarzik>
jonasschnelli, sqlite3 branch is at a temporary end state, unless further speedups can be found
< jgarzik>
jonasschnelli, I task switched over to BIP 1xx and pgdb2
< sipa>
jonasschnelli: #6948 is only expected to solve (or improve at least) the undo corruption, not leveldb middle of record stuff
< jonasschnelli>
sipa: Right. I see. I've thought together with #6917 (leveldb) it would fix the whole corruption issues. Because with 6917 i "mostly" got the "bad undo" error... but now i get 100% leveldb middle of records errror.
< sipa>
are you sure you did #6948 together with #6917?
< sipa>
and not only #6948?
< GitHub50>
[bitcoin] morcos closed pull request #6857: Require nLastBlockFile to be the highest numbered file. (master...nLastBlockFile) https://github.com/bitcoin/bitcoin/pull/6857
< jonasschnelli>
anyone else facing qt5 depends build issues when building with mingw on debian or ubuntu?
< GitHub113>
[bitcoin] MarcoFalke opened pull request #6951: [qt] Use maxTxFee instead of 10000000 (master...MarcoFalke-2015-qtMaxFee) https://github.com/bitcoin/bitcoin/pull/6951
< gmaxwell>
jtimon: I am not suggesting it for using it with blocks. I think it would be bad for that.
< gmaxwell>
wumpus: for all you know there is something wrong with jonasschnelli's VM and it impermisably reorders writes or ignores sync.
< gmaxwell>
wumpus: (I would be completely unsurprised to see VM software which ignored sync to prevent vm guest OS from thrashing the host... and used userspace buffering of writes for similar reasons, and didn't guarentee them flushed when it quit)
< gmaxwell>
wumpus: in any case, there is a clear path forward, get the corrupted data from him and check it out.
< gmaxwell>
wumpus: based on the undo related errors before I believe his node is in the process of a reindex as he's doing these tests?
< gmaxwell>
If it isn't the VM itself that is the difference in your test conditions, then it may be reindexing.
< wumpus>
true, that may be the case
< wumpus>
going to test master+#6948 on real-windows tomorrow, if I can't get corruption, I'll just ignore the VM results for now
< jgarzik>
some VMs definitely do that
< jonasschnelli>
Okay. This might be the case (although I never expected any corruption while hard power off a VM).
< GitHub112>
[bitcoin] TheBlueMatt closed pull request #6595: Fix removal of timelocked-txn from mempool during reorg (master...lockedmempool) https://github.com/bitcoin/bitcoin/pull/6595
< jtimon>
gmaxwell: I know, you are suggesting to use it for acceptToMemoryPool