< Hisham>
At first, it's my pleasure to talk to you :)
< Hisham>
If you don't mind, I would like to discuss something with you regarding SegWit UASF
< Hisham>
Can we talk please ?
< BlueMatt>
dear god, dont ask to ask, just ask
< Hisham>
:S
< Hisham>
Guys, I know you are nice to people, c'mon, what happened ?
< Hisham>
Maybe you are busy or something, does politeness and taking permission is considered a crime nowadays ?
< BlueMatt>
it is generally discouraged on irc
< BlueMatt>
text-based communication is slow enough already :(
< Hisham>
Ok I see and understand, no problem :)
< BlueMatt>
engineers, you know
< Hisham>
Here is one as well, Mechanical and a CNC programmer :)
< Hisham>
I was going to be a colleague, but I had a nasty doctor who let me hate CS, Java to be precise.
< gmaxwell>
google for "don't ask to ask"-- beyond being wasteful, it's considered rude because it asks people to commit to an open ended conversation which they may not be interested in when they know the details.
< gmaxwell>
as far as your question, all the discussion on that has been on the mailing list, I don't think anyone here has been involved in it (though perhaps I'm incorrect, as I haven't been following it)
< sipa>
Hisham: just ask whatever you want to ask
< Hisham>
Thanks for the clarification Gregory.
< Hisham>
But guys, for God sake take it easy on me.
< sipa>
yes, get to the point :)
< Hisham>
I will.
< Hisham>
I'm a normal old Bitcoiner since the days of CPU and browser mining but I've got highly involved at 2014, I've read a lot about Bitcoin and you can I'm my life is oriented to it for many reasons.
< Hisham>
When I saw the blocksize debate, I felt that I can't stand on the sidelines and try to help or at least understand and engage in conversations, maybe I can do something for all the BS happening around us.
< achow101>
Hisham: please just ask your question. none of this preamble is necessary. I hate to break it to you, but no one cares about your backstory. Please just get straight to the point and ask your question.
< Hisham>
No problem, I understand, sorry.
< Hisham>
As some of you may witness, Shaloin Fry made what we can call a SegWit UASF.
< Hisham>
As I understand, it needs consensus, if I'm not mistaken.
< Hisham>
How can we get this implemented if there is a chance for it to be implemented ?
< sipa>
no politics here, please
< BlueMatt>
Hisham: if you want a UASF to happen, go start talking to the community - talk to businesses and users
< BlueMatt>
such things are not for us to decide
< BlueMatt>
Hisham: relevant is bluematt.bitcoin.ninja/2017/02/28/bitcoin-trustlessness/ (and also #bitcoin, not really #bitcoin-core-dev)
< * BlueMatt>
does the shameless-self-promotion dance
< Hisham>
I understand, but from technical POV, is there a chance for intergation ?
< Hisham>
*integration ?
< sipa>
implementing it is absolutely trivial
< sipa>
the only question is whether we should
< sipa>
and generally the answer to that is "if it's clearly uncontroversial"
< BlueMatt>
i believe code was already written, as for people actually using that code....<BlueMatt> such things are not for us to decide
< Hisham>
Pieter, may you elaborate more please on this "if it's clearly uncontroversial"
< Hisham>
?
< BlueMatt>
maybe this is a better topic for #bitcoin, its not really a development topic
< Hisham>
Ok, since I'm not welcomed form the beginning.
< BlueMatt>
Hisham: I'm happy to go discuss it further on #bitcoin
< Hisham>
Ok, can we talk there now please along with Pieter ?
< bitcoin-git>
bitcoin/master fd5e905 Matt Corallo: Make verify-commits.sh non-recursive
< bitcoin-git>
bitcoin/master 8ed849f Matt Corallo: Fix travis failing to fetch keys from the sks keyserver pool...
< bitcoin-git>
bitcoin/master efc06c2 Matt Corallo: If GNU sha512sum is missing, try perl shasum in verify-commits
< bitcoin-git>
[bitcoin] laanwj closed pull request #9940: Fix verify-commits on OSX, update for new bad Tree-SHA512, point travis to different keyservers (master...2017-03-verify-commits-no-recursion) https://github.com/bitcoin/bitcoin/pull/9940
< bitcoin-git>
[bitcoin] laanwj opened pull request #9983: tests: Convert selected tests to using named RPC arguments (master...2017_03_rpc_tests_named_arguments) https://github.com/bitcoin/bitcoin/pull/9983
< wumpus>
okay, yeah I don't really like that solution. It should hash what is in git, not what is in the current working directory
< MarcoFalke_lab>
ok, fine.
< wumpus>
otherwise there's always a window for races
< MarcoFalke_lab>
jup, it is the cleaner solution.
< wumpus>
working on it now - apparently ls-files gives the blob ids, that can be passed as-is to git cat-file
< BlueMatt>
wumpus: yes, iirc that was annoying but i dont remember why now....verify-commits does cat-file $COMMIT_ID:$FILE_PATH
< wumpus>
BlueMatt: ah, that likely works just as well
< BlueMatt>
wumpus: if you can get it to work some other way, though, use that...I kinda liked how merge did the hashes via checkout and verify-commits used a different approach, but it really doesnt matter all that much
< * BlueMatt>
doesnt have to deal with the merge script, so I get to argue for it to be harder to use all day long with no repurcussions, though :P
< wumpus>
well how the sha is computed doesn't affect how hard it is to use, it does affect whether the output is correct, which is arguably very important
< BlueMatt>
indeed
< * BlueMatt>
is ok with the sha being less likely to match, if that also means its more likely to trip up if something strange happens (symlink detection gets broken, etc)
< BlueMatt>
anyway, this is too inconsequential to nitpick over :P
< wumpus>
I'm kind of annoyed at travis tripping up all the time though, if that keeps happening I'll hvae to disable the verify-commits check there. SO let's please just get it working
< BlueMatt>
i think it should be good now
< BlueMatt>
it failed a number of times just downloading the pubkeys because the sks pool was broken
< BlueMatt>
but we changed to subset which appears to possibly be more stable
< BlueMatt>
i do not believe we have otherwise failed in some time
< BlueMatt>
hmmm, no, failed again this morning
< BlueMatt>
I'll go take a look
< wumpus>
according to MarcoFalke_lab it failed due to my merge script miscomputing the SHA
< MarcoFalke_lab>
BlueMatt: It used the 'old' server url
< MarcoFalke_lab>
The other failure was due to the wrong sha
< BlueMatt>
that was once, looks like it also once failed to retreive pubkeys
< BlueMatt>
wtf
< BlueMatt>
how did it do that? it now points to subset???
< MarcoFalke_lab>
BlueMatt: IIRC the subset-patch was not merged at that time
< BlueMatt>
yes it was, the very top commit failed and the second-to-top merge was the subset change
< wumpus>
what subset patch?
< BlueMatt>
im super confused
< BlueMatt>
wumpus: part of #9940 was to change the server gpg fetches the keys from
< gribble>
https://github.com/bitcoin/bitcoin/issues/9940 | Fix verify-commits on OSX, update for new bad Tree-SHA512, point travis to different keyservers by TheBlueMatt · Pull Request #9940 · bitcoin/bitcoin · GitHub
< wumpus>
ok, confusing
< MarcoFalke_lab>
Right
< MarcoFalke_lab>
Probably just a travis network hickup
< petertodd>
BlueMatt: why not have the merge script get the pubkeys from the repo itself?
< BlueMatt>
MarcoFalke_lab: no, look at the build, it tried to fetch from pool., not subset.pool.
< BlueMatt>
petertodd: yea, I think thats the next step, I was thinking subset might help, but...
< BlueMatt>
still need to do a --refresh-keys
< BlueMatt>
but need a backup
< petertodd>
BlueMatt: better to make the whole thing deterministic; if verify commits doesn't run because a key is missing, fix that!
< BlueMatt>
oh, no, I'm wrong, github is listing me commits out of order
< BlueMatt>
wtf
< wumpus>
yes it does that, it uses a stupid sort ordering for commits
< wumpus>
by date instead of chronologically according to the repo
< BlueMatt>
oh, ok, yes, the failure occurred before the subset merge, and then another afterwards due to bad sha512
< BlueMatt>
wumpus: well i was only looking at merge commits, i think github actually just had a hiccup
< BlueMatt>
anyway, I restarted the build on the >=8 -addnode fix merge, which should pass this time, if there are any non-sha512-related failures we'll switch to pulling keys from in-repo
< wumpus>
argh, git cat-file blob <blob> is slow compared to the current script
< wumpus>
oh it can work in batch mode, that's cool
< BlueMatt>
wumpus: its hella slow
< BlueMatt>
like, insanely slow
< wumpus>
also in batch mode? I think the overhead right now is spawning a process for every file
< BlueMatt>
hmm, not sure, didnt try, but you need to separate out different files for our current hash format
< BlueMatt>
argh, great, now i need to make verify-commits use batch mode
< BlueMatt>
:P
< bitcoin-git>
[bitcoin] laanwj opened pull request #9984: devtools: Make github-merge compute SHA512 from git, instead of worktree (master...2017_03_merge_hash_git) https://github.com/bitcoin/bitcoin/pull/9984
< bitcoin-git>
[bitcoin] NicolasDorier opened pull request #9985: [QT] Show more descriptive label for pay to yourself entries (master...watchonlylabel) https://github.com/bitcoin/bitcoin/pull/9985
< gmaxwell>
Apparently IBD on 0.14 OOMs on 2GB (at least on an odroid c2, as expirenced by sipa and someone on reddit); due to the mempool loaning allowing it to achieve peak memory at IBD rather than sometime later.
< gmaxwell>
I suppose it's better to crash during IBD rather than later...
< achow101>
gmaxwell: someone has reported that in an issue and on bitcointalk too
< gmaxwell>
achow101: well the answer for them right now is to lower their dbcache/maxmempool, I recommended 150/150.
< gmaxwell>
It's not really a regression in 0.14, the same devices would eventually crash with 0.13.x, just later.
< sipa>
well, not necessarily, but they would with -dbcache=600
< sipa>
so we've effectively just raised the default dbcache during IBD
< gmaxwell>
sipa: say dbcache is 300, and mempool is 300.. eventually dbcache will be full and mempool will be full.... sooo similar peak memory usage, no?
< gmaxwell>
(or would you argue that because the leveldb batch can result in 2x memory, that we should be adding half the mempool to the dbcache?)
< sipa>
gmaxwell: right, that's what i'm saying
< sipa>
with 300 MB dbcache + 300 MB mempool, you can use up to 900 MB at flush time
< sipa>
with 600 MB dbcache + 0 MB mempool, you can use up to 1200 MB at flush time
< gmaxwell>
so perhaps a patch we should do quickly and backport is to add half the unused memory to the dbcache. Can you see if that would make your odroid c2 successfully sync?
< sipa>
BUT, we should probably halve the -dbcache setting as well, because someone who configures 2300 MB of dbcache does not expect the application to suddenly use 4600 MB, regardless of mempool borrowing
< gmaxwell>
hm. so perhaps half the units, double the default?
< gmaxwell>
halve*
< sipa>
or just fix leveldb to not require the whole batch to be stored in a single std::string.
< gmaxwell>
yea, but would we backport that for 0.14.1?
< sipa>
depends on how trivial it is, i think
< gmaxwell>
(I think we need to do something for 0.14.1 but it could be something dumb like halving the mempool loan)
< sipa>
yeah, that would fix the 'unexpectedness' of the memory loaning
< sipa>
but not the incorrectness of the estimate
< luke-jr>
is there a reason to use dbcache post-sync?
< sipa>
yes, much faster block verification
< sipa>
even with sigcache
< luke-jr>
but why not commit it after each post-sync block immediately?
< luke-jr>
it doesn't grow that much from just one block, does it?
< sipa>
it can grow by 20 MB or so from one block
< sipa>
and flushing slows down future blocks
< luke-jr>
hmm
< sipa>
with per-txout caching, the max (and typical) growth of the cache per block will be much less
< gmaxwell>
much (most?) of dbcache's gains come from avoiding writes entirely for utxo that never need to make it to disk.
< sipa>
indeed.
< sipa>
though avoiding writes can always be moved out of the critical path of block validation
< sipa>
in synced state, it still prevents recent utxos from needing to be read from disk
< nemgun>
Hello guys, thank you for 0.14, i am resyncing, and it uses far less CPU
< luke-jr>
sipa: aside from validation, that's probably useful for external tools accessing them via RPC?
< sipa>
luke-jr: i believe gettxout will use the cache
< sipa>
luke-jr: gettxoutsetinfo always flushes to disk, and then iterates over the leveldb state
< TD-Linux>
<gmaxwell> I suppose it's better to crash during IBD rather than later... <- yeah though super annoying when pruning.
< gmaxwell>
oh my we should probably only be pruning after sync flushes.