< fanquake> Anyone want to throw an ACK onto #13093, pretty trivial backport to get 0.15 Qt depends building again on macOS
< gribble> https://github.com/bitcoin/bitcoin/issues/13093 | [0.15] backport: depends qt patches by fanquake · Pull Request #13093 · bitcoin/bitcoin · GitHub
< murrayn> utACK, I'm not able to test it.
< promag> #13160 <- is this considered bug fix? also backport? needs release note?
< gribble> https://github.com/bitcoin/bitcoin/issues/13160 | Unlock spent outputs by promag · Pull Request #13160 · bitcoin/bitcoin · GitHub
< bitcoin-git> [bitcoin] martinus opened pull request #13176: Improve CRollingBloomFilter performance: replace modulus with FastMod (master...optimize-CRollingBloomFilter) https://github.com/bitcoin/bitcoin/pull/13176
< booyah> -Copyright (C) 2009-2017 The Bitcoin developers
< booyah> -Copyright (C) 2009-2017 The Bitcoin Core developers
< booyah> +Copyright (C) 2009-2018 The Bitcoin developers
< jojeyh> am i correct in assuming that the "bestblock" field in the output from `bitcoin-cli gettxout` is the block hash of the block that includes the "txid" passed into the argument ?
< jojeyh> if i am, i think something is wrong
< jojeyh> the block hash printed out does not match the block where the txid is
< provoostenator> fanquake: done. You may return the favor by testing #12404 :-)
< gribble> https://github.com/bitcoin/bitcoin/issues/12404 | Prune more aggressively during IBD by Sjors · Pull Request #12404 · bitcoin/bitcoin · GitHub
< provoostenator> I also got the bright idea of making my ccache larger (~/.ccache/ccache.conf) and moving it to my SSD drive...
< bitcoin-git> [bitcoin] ken2812221 opened pull request #13177: GCC-7 and glibc-2.27 compat code (master...compat) https://github.com/bitcoin/bitcoin/pull/13177
< bitcoin-git> [bitcoin] donaloconnor opened pull request #13180: Fix re-declared scoped enum as unscoped (Causes issues with some compilers) (master...06052018_unscoped_enum) https://github.com/bitcoin/bitcoin/pull/13180
< provoostenator> Am I correct that during IBD the bulk of random read/write disk activity is for /blocks/index and /chainstate? Whereas /blocks gets written in larger batches?
< sipa> eh
< provoostenator> Totally wrong?
< sipa> blocks are written to more frequently (every block), where index/chainstate only when the cache fills up
< sipa> blocks and index are generally never read
< sipa> (blocks are read for rescan and serving to peers; index is only read at startup)
< sipa> chainstate is read/write and sorta random access, due to leveldb most of the writes are sequential
< provoostenator> For context, I'm trying to figure what the ultimate IBD EC2 instance would look like, in order to produce a small pruned result, which I would then copy to a low-end machine. Particularly what needs a RAM drive.
< sipa> much RAM
< sipa> run with large dbcache
< sipa> if your dbcache is 6G or so, there are no ransom access reads or weites at all
< provoostenator> The x1e.2xlarge has 244 GB RAM, so I can store the whole chain. But if > 6 GB has extremely dimishing returns I'd rather get more CPU's probably.
< sipa> yes
< sipa> get 8G or so
< sipa> storing the chain itself in RAM is useless
< sipa> it's not accessed during sync
< provoostenator> Has anyone ever plotted IBD vs. number of CPU's? I assume that flattens off at some point. EC2 can do up 72 CPUs :-)
< sipa> bitcoin core won't even use more than 16
< sipa> lock contention because worse than the gains
< sipa> i doubt it matters much if you use more than 8
< provoostenator> And it doesn't use GPU, right?
< sipa> nope
< provoostenator> But a local SSD drive might? Alright, I picked four machine with 8 CPU's each and some differences otherwise: t2.2xlarge, m5.2xlarge, c5.2xlarge and i3.2xlarge. I'll let them race eachother tonight...
< sipa> if your dbcache is big enough, disk doesn't matter at all
< sipa> you're configuring dbcache as large as possible on each?
< provoostenator> Oh ok, I'll stick to the first three then (with massive dbcache).
< sipa> it won't automatically use more than 450M unless instructed
< sipa> numbers would be interesting, though
< provoostenator> Any branch other than master you're interested in?
< sipa> if dbcache is not sufficiently large to fit the entire sync in ram, random access read speed of disk does start mattering
< provoostenator> These machines all have about 60 GB of RAM, so I'll set -dbcache=50000
< provoostenator> When I did my benchmarking last year I never ticked the "allow burst above baseline" options, so I probably got nowhere near max CPU performance.
< gmaxwell> provoostenator: EC2 has terrible terrible terrible IO latency in general.
< gmaxwell> and it slaughters performance unless your dbcache is big enough for the whole UTXO set (perhaps around 6GB or so)
< gmaxwell> (similar for most other VPS providers)
< provoostenator> Amazon seems to assume that most applications only need short CPU burst and so they make you "save" credits for these incidental burst. But obviously IBD is bit more intense.
< sipa> well verify that it isn't actually flushing
< provoostenator> How do I reduce the log output to only show flushes?
< sipa> (you'll see the cache size field in the UodateTip line drop to ~0)
< gmaxwell> you can tell from the updatetip lies.
< gmaxwell> er lines
< provoostenator> :-)
< sipa> (sorrry for typos, in a plane, bad lag)
< provoostenator> Clock started at 20:30 UTC. The t2.2xlarge just crossed the 200K block line and is slightly ahead of the other two.
< spudowiar> provoostenator: Do one with Bitcoin-ABC :P
< spudowiar> I would say do Ethereum but... :)
< provoostenator> Someone else is welcome to do that. Apparantly Etherehum is cloud resistant, according to a recent Noded podcast with Jameson Lopp. Due to relatively heavy I/O, if you fully validate.
< spudowiar> Apparently Ethereum is validation-resistant
< spudowiar> I wonder how Anduck's sync is getting on
< sipa> he gave up afaik, but this is getting pretty off topic
< * spudowiar> is a bad influence :)
< provoostenator> Indeed, I'll post the result tomorrow morning.
< spudowiar> provoostenator: There's no networking difference between instances, is there?
< provoostenator> I don't think networking is a bottleneck with these machines, but it could explain small differences in the beginning. Things don't get interesting until ~2015 when the UTXO set grows.
< Anduck> spudowiar: my machine ran out of space for non-pruned eth sync, got frustrated and haven't since done it again. it synced at some pace (my estimation is 4-5 weeks for full sync) with 3x ssd in raid0, with hdd would be impossible.
< spudowiar> nice
< Anduck> this is the 2nd test. initial one i gave up as it was clearly impossible to never get it synced
< jojeyh> i had to stop running an eth node cuz it made my laptop completely unusable
< phantomcircuit> provoostenator, you cant actually fully validate ethereum anymore
< phantomcircuit> there's a stretch of blocks which are effectively impossible to validate on anything
< phantomcircuit> besides they just make up the rules randomly anyways
< provoostenator> phantomcircuit: let's not go too far off topic. I'll post the result here, but unless there's a PR that can improve it, this is probably not the right place.
< provoostenator> Just created #blockchain-sync, let's move there.
< bitcoin-git> [bitcoin] donaloconnor closed pull request #13180: Fix re-declared scoped enum as unscoped (Causes issues with some compilers) (master...06052018_unscoped_enum) https://github.com/bitcoin/bitcoin/pull/13180
< bitcoin-git> [bitcoin] donaloconnor reopened pull request #13180: Fix re-declared scoped enum as unscoped (Causes issues with some compilers) (master...06052018_unscoped_enum) https://github.com/bitcoin/bitcoin/pull/13180