< bitcoin-git>
bitcoin/master 6c5427d Wladimir J. van der Laan: wallet: Prevent "overrides a member function but is not marked 'override'" warnings...
< bitcoin-git>
bitcoin/master f8af89a Wladimir J. van der Laan: Merge #9764: wallet: Prevent "overrides a member function but is not marked 'override'" warnings...
< bitcoin-git>
[bitcoin] laanwj closed pull request #9764: wallet: Prevent "overrides a member function but is not marked 'override'" warnings (master...2017_02_wallet_inconsistent_missing_override) https://github.com/bitcoin/bitcoin/pull/9764
< wumpus>
I don't know. I'd say let's not add more stuff now and try to get a rc1 wrapped up
< wumpus>
it's a release candidate to get more testing
< wumpus>
issues will always be found in RCs of major releases
< luke-jr>
sure
< gmaxwell>
9524 looks harmless.
< wumpus>
we can keep pushing this forward indefinitely and just not do releases anymore :/
< luke-jr>
not saying it needs to get into rc1, just hoping to avoid thigns being overlooked before final
< gmaxwell>
wumpus: would solve so many problems!
< gmaxwell>
luke-jr: I don't think that should go into 0.14.0, unless it has some effect I'm missing.
< wumpus>
it's under-discussed and under-reviewed
< gmaxwell>
by harmless I mean don't do it.
< gmaxwell>
The question is "what bad expirence will the user or network have as a result of this not going in" if the answer is none we shouldn't even be remotely considering it now.
< gmaxwell>
I mean the thing it fixes does not meet the above test ^, sorry for being unclear.
< wumpus>
oh I agree then :)
< luke-jr>
pruneblockchain(0) would probably delete a lot of data the user doesn't intend
< gmaxwell>
okay, that would be bad.
< gmaxwell>
wait, the argument there is what, /me looks
< luke-jr>
at least as I understand it, the 0 indicates the node should automatically prune up to tip minus saved-amount
< wumpus>
your are misunderstanding it luke-jr
< wumpus>
argument 0 means *do nothing*
< wumpus>
it doesn't delete anythhing in that case
< wumpus>
it just flushes to disk
< wumpus>
who cares :)
< gmaxwell>
That was what I got out of the PR.
< wumpus>
that PR bypasses the flush, I didn't regard that as very important as it's a NOOP anyway
< gmaxwell>
If it did what luke said-- prune as hard as allowed--, that would be worth fixing before final.
< luke-jr>
what am I missing?
< wumpus>
what you are missing is that pruneblockchain(0) already does nothing besides a FlushToDIsk
< luke-jr>
FlushStateToDisk with 0 triggers FindFilesToPrune(setFilesToPrune, chainparams.PruneAfterHeight()); etc
< wumpus>
that PR takes away the FlushToDisk
< wumpus>
"It might cause unwanted bugs if someone refactors the code in the future."
< wumpus>
that's all
< wumpus>
which we can consider for 0.15, but there will be no refactors of that code in the future of 0.14, so it's pointless there
< * gmaxwell>
risks his laptop node.
< gmaxwell>
luke-jr: so just calling that with 0 throws cannot prune because not in prune mode.
< luke-jr>
what manual pruning is enabled?
< gmaxwell>
trying.
< gmaxwell>
pfft. it's yelling at me because of txindex. @#$@
< luke-jr>
>_<
< gmaxwell>
so to do that test I'd have to reindex which would probably take days on this thing. :-/ and my other nodes are busy now with important test.s
< gmaxwell>
I will test before 0.14 release however, if no one else does.
< Victorsueca>
need testing?
< gmaxwell>
if you don't mind blowing away your blockchain if luke is right.
< wumpus>
ideally *disabling* txindex would not require a rescan, but meh :) I'll test it
< * wumpus>
well test, has about 50 copies of the blockchain anyhow
< gmaxwell>
Start a node without txindex with prune=1, then run pruneblockchain 0 and verify all your blocks didn't just suddenly go away (e.g. freeing up 100 GB of disk space).
< Victorsueca>
ummm, I'll make a backup of my current chain, it probably won't be in time to test this one, but i'll be ready for future chain-blowing PRs
< wumpus>
possibly no need to even copy, would hard-linking all but the latest block file work?
< luke-jr>
not sure we have a way to restore such a backup without the chainstate
< wumpus>
okay, definitely not going to try that at the same time
< wumpus>
luke-jr: why is it a problem in this case to copy the chainstate too?
< luke-jr>
wumpus: it might not be, just saying in case copying it also wasn't obvious
< gmaxwell>
FS with cow would be nice for some of these tests.
< * luke-jr>
does have a CoW filesystem, but also has txindex enabled etc.
< wumpus>
luke-jr: if you don't copy the chainstate, all it can do is a reindex
< luke-jr>
I suppose a unit test could check this, but seems probably too bug-specific
< luke-jr>
s/unit/rpc/
< wumpus>
well if this turns out to be a regression then a regression test could be added for it, but if this never was a bug in the first place...
< luke-jr>
0.13 didn't support manual pruning afaik
< wumpus>
then we're just chasing ghosts
< wumpus>
though testing it in regtest instead of a live node is a very good idea
< Victorsueca>
so, if I copy the blocks and chainstate folder it should be enough right?
< wumpus>
yes
< luke-jr>
Victorsueca: may need to be sure the node is stopped when you copy
< Victorsueca>
ok
< wumpus>
oh absolutely, don't do anything low-level with the data files without stopping, that shouldn't have to be said
< Victorsueca>
yeah it's stopped, the windows that tells me to wait disappeared too
< wumpus>
on windows, copying *from* the bitcoin folder while bitcoin core is running can even cause it to crash
< wumpus>
not sure why, I'm not aware ldb does anything special while opening the files
< Victorsueca>
maybe something related with atomic locks?
< * luke-jr>
found it curious to see a Windows user actively using a "file unlocker" program like it was normal
< Victorsueca>
windows file locking is great at preventing programs from crashing due to files that where expected to be there and now are not
< Victorsueca>
but it's stupid in most cases
< wumpus>
yes, it's great at preventing programs from crashing... we see that :-)
< Victorsueca>
lol
< Victorsueca>
let's just say the remedy is worst than the ill
< Victorsueca>
it could be way better implemented
< wumpus>
it's probably not that bad if you take extensive time to study the APIs involved and use them as they are supposed to be, but who has time to do native development for every special snowflake platform :/
< midnightmagic>
pure snapshotting would work just fine for this, if you could convince the software to lock all files prior to the snapshot, or guarantee a write barrier
< wumpus>
the annoying thing with microsoft has always been that they take existing concepts then change them slightly, just enough to be incompatible and lock you in a bit
< wumpus>
POSIX? forget it
< midnightmagic>
windows file locking works fine for monolithic files that are only modified by the software modifying them, even in 10,000+ user database environments far busier than I suspect bitcoin's databases will ever be.
< luke-jr>
I thought Windows was POSIX compatible?
< midnightmagic>
Windows has POSIX emulation layers but they all operate differently than UNIX. For example, when a file is locked, it is a much harder lock than its POSIX-style equivalents on other systems.
< wumpus>
it tries, a bit, the devil is in the details
< midnightmagic>
You can delete a file which is locked, which zeroes it, but you can't subsequently recreate a file with the same *name*.
< midnightmagic>
(for example)
< luke-jr>
O.o
< midnightmagic>
(external to the program which has the file locked)
< midnightmagic>
There are other operations which are virtually impossible to guarantee the atomicity of, even with the POSIX compatibility layer because all the things we take for granted are emulated badly, as wumpus implies.
< Victorsueca>
I think the problem is at assertions
< Victorsueca>
windows file lock asserts a lo of thing that are not always true
< Victorsueca>
lot*
< wumpus>
yes it's not that windows's way doesn't work, or even works worse in the abstract sense, it's just that they force you to develop platform specific code for everything
< wumpus>
they did that with directx/opengl, with winsock, and the list goes on
< midnightmagic>
Meanwhile, on Windows it's possible to hook file open calls with e.g. realtime virus, but contrary to anti-virus checkboxes, it's actually not possible to apply it selectively to a path and not other paths. It's all-or-nothing. That's why basically anyone running a database with an active realtime virus checker is sitting on a dice-rolling timeline. Eventually, their database will be
< wumpus>
anyhow, I was testing pruning
< midnightmagic>
destroyed. And there's absolutely nothing anybody can do about it.
< wumpus>
okay, tested: pruneblockchain 0 does nothing and returns immediately.
< wumpus>
Arguments: "height" (numeric, required) The block height to prune up to. May be set to a discrete height, or to a unix timestamp to prune based on block time.
< Victorsueca>
wumpus: was that on linux or windows?
< wumpus>
linux
< * luke-jr>
wonders why
< Victorsueca>
is it useful if I test it on windows now or it's about the same for this case?
< wumpus>
well 0 is the genesis block, so pruning from there means not pruning at all
< wumpus>
no, it's not useful to test this on other OSes
< luke-jr>
wumpus: yes, but the code special-cases 0 as automatic
< Victorsueca>
maybe that automatic check thought no pruning was needed at all? or that's not how it works?
< wumpus>
GAH I can't get manual pruning to work at all on regtest. I created a 10000 block chain, started with prune=1, and no matter what I call pruneblockchain with it does nothing. So I guess my test is invalid :/
< wumpus>
oh shit of course, it will only prune once it gets to another block file
< luke-jr>
>_<
< wumpus>
okay, generating a ton more bloocks
< wumpus>
maybe using a live chain wasn't that bad an idea afterall, I now realize
< gmaxwell>
The help text repeadily uses 1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XZ as an example address. In general I think it's unwise to have valid example addresses, but I can't find any information about what address it is. It's a real working address that has been paid .2488 btc and has spent payments.
< gmaxwell>
the commit that added it and the related PRs seem to make no mention of the address.
< gmaxwell>
added by commit a6099ef319a73e2255dca77065600abb22c4f5f8
< wumpus>
bleh, I thought we got rid of that
< gmaxwell>
might be gone now, lemme look
< wumpus>
no it's still there
< gmaxwell>
nope, still there.
< wumpus>
I vaguely remember that it was replaced with a constant defined in one place, which was not a valid address
< wumpus>
but that must be on another timeline...
< gmaxwell>
I had thought some charity ('sean's outpost') address was used (which I also think is not great) but I don't see any reason why I might have believed that this was that.
< gmaxwell>
wumpus: yea, I too spend a lot of time in alternative universes.
< wumpus>
hehe :) oh I think I know, that was for the GUI.
< Victorsueca>
yeah, the e.g. on the pay to field got changed
< gmaxwell>
it got put as an example in the pay to field? 0_o
< Victorsueca>
I vaguely remember some PR requesting to change it by valid looking address that is not really valid
< Victorsueca>
I'm using 0.13.2 and now on that field there is 1NS17iag9jJgTHD1VXjvLCEnZuQ3rJDE9L
< gmaxwell>
yep, not valid, which is what it should be IMO.
< wumpus>
yes, a non-valid address was put in the pay-to field. That address, too, used to be valid at some time. Although it was terribly hard to type it over because it disappears when you start typing into the field. The RPC example is worse.
< midnightmagic>
30 helens agree
< wumpus>
it reminds me of a dumb mistake I made when I was young, a computer manual had KILL *.* as example for the KILL statement, which deleted. So while trying around I typed that in. Oops, that wiped the entire floppy disk, my dad was angry and I felt really stupid. At least that would have been recoverable, if I knew back then...
< midnightmagic>
lol
< gmaxwell>
I could see someone getting confused and copying from the wrong line on their screen... one address looks like any other..
< wumpus>
yes it's pretty easy to copy the entire example
< midnightmagic>
Who's "sje"?
< wumpus>
at least the only sending call where it's used is sendmany, and it only sends relatively small amounts
< wumpus>
still it should be replaced with a constant that is not a valid address, likein the GUI
< wumpus>
so while juggling those things, memory became lossy :)
< wumpus>
gmaxwell: LOL
< wumpus>
gmaxwell: that won't work, your address is not in any RPC examples
< wumpus>
anyhow, ready to re-retry the pruning experiment with a real mainnet blockchain
< gmaxwell>
AND ITS <GONE / NOT GONE>.
< wumpus>
it's doing nothing and logging nothing, so my would be "NOT GONE"
< wumpus>
genesis block is still ther
< MarcoFalke_>
wumpus: You need to revert ntl. The commit is not signed :)
< wumpus>
I also understand why @luke-jr - in FlushStateToDisk, it only goes into the pruning flow if fCheckForPruning || nManualPruneHeight > 0 ... fCheckForPruning is not set in manual pruning mode
< wumpus>
huh?
< wumpus>
I really don't get it now, this was using the script, it shouldn't push if not signed
< * wumpus>
isn't sure if he's becoming crazy or his tooling is
< MarcoFalke_>
maybe you `git push bitcoin` in another tab?
< wumpus>
and back to e43a58514dd38dacd930aa4c94afb998d4360183
< MarcoFalke_>
`git commit --amend` gets rid of the signature
< wumpus>
the github-merge tool should probably get a check that what it is pushing is really what it expects to be pushing, in case the current branch changed under it
< wumpus>
or rather, push a specific commit, not what happens to be the current branch
< wumpus>
not sure it does that
< gmaxwell>
oh did you run the merge tool twice concurrently?
< wumpus>
blah, I wasn't intending to dive into source code management specifics today
< wumpus>
gmaxwell: that's possible!
< gmaxwell>
"This worksite has gone [ 0 ] days since an SCM emergency."
< Victorsueca>
any PR that would be useful to test at the moment?
< morcos>
wumpus: all for 0.14rc1 today but i think we need to wrap up the importmulti issues... ryanofsky opened #9773 yesterday to try to address the combination of what you and gmaxwell were asking for
< gribble>
https://github.com/bitcoin/bitcoin/issues/9773 | WIP: Return errors from importmulti if complete rescans are not successful by ryanofsky · Pull Request #9773 · bitcoin/bitcoin · GitHub
< morcos>
i tihnk #9671 should also be included (it's simple and could avoid fund loss)
< wumpus>
morcos: my brain is not working today so I've already given up on that
< morcos>
heh fair enough.. but still need to get those 2 across the finish line
< wumpus>
9671 is already merged?
< morcos>
sorry #9761
< gribble>
https://github.com/bitcoin/bitcoin/issues/9761 | Use 2 hour grace period for key timestamps in importmulti rescans by ryanofsky · Pull Request #9761 · bitcoin/bitcoin · GitHub
< morcos>
i think you are contagious
< wumpus>
I'm okay with adding new things if they're ready for merge, everything that requires new review should probably be pushed to after 0.14.0, at some point we have to make a cut
< wumpus>
I really can't keep moving the 0.14.0 rc1 day after day forward, keeping the tree in this state is holding up more and more things that could be merged for 0.15
< wumpus>
but one more day is fine, as said, I'm not up to it today anyhow
< morcos>
i don't think ryanofsky or i have a strong opinion about it, it seems you guys were the ones saying the current state is not acceptable
< morcos>
current state is silent failure when you do importmulti on pruned node that needs to search more blocks than you have
< wumpus>
at some point you should think of it as "is it acceptable for a rc1"?
< wumpus>
it's certain that issues come up with user testing and have to be solved before rc2
< morcos>
we're just trying to fix the problems
< wumpus>
sure, and that's very good, but there's always new problems
< morcos>
but would be helpful to have you and gmaxwell take a look and say yes thats what i wanted
< wumpus>
yes the approach in 9773 looks good to me
< fanquake>
Looks like you can't re-open a merged PR.
< mryandao>
i'm having a bit of problem after updating my tree from upstream
< mryandao>
this is the error i'm getting
< mryandao>
util.cpp:834:63: error: use of undeclared identifier 'COPYRIGHT_HOLDERS'
< wumpus>
I'd prefer a small bar with a few icons - making it big like this just takes up extra screen space in a site that should be focused on the code
< Victorsueca>
wumpus: you talking about that new black bar at the top?
< fanquake>
Indeed. Especially when your on a smallish screen.
< wumpus>
Victorsueca: yep
< wumpus>
fanquake: indeed. Well even big modern screens are usually wide, but not very high, so vertical screen space is at premium
< wumpus>
fanquake: will take a look, thanks
< Victorsueca>
for some reason it becomes white if you logout
< Victorsueca>
but yeah, it's ugly, it takes much attention and distracts people from important things
< Victorsueca>
it's too big and contrasted
< wumpus>
indeed
< wumpus>
9763 is a no-brainer, I can do that one, at least if I get the numbers right :p
< wumpus>
looks like it could use squashing though
< fanquake>
Iwumpus 'm also vary calling anything an "easy" merge, without a good way to check if they somehow break other more-important patches.
< fanquake>
*I'm always
< wumpus>
three commits to make three related documentation-only changes
< fanquake>
GitHub doesn't seem to have a way to check, if I merge this, what other PR's will it break. Would be handy when trying to cehck quickly.
< wumpus>
not sure if I'm up to such "advanced" git manipulation at the moment... let's see
< wumpus>
fanquake: agree, that would be great to have
< wumpus>
fanquake: at a company I worked at they had a script to generate a 'patch PERT' which was a diagram of which things could be merged without (code level) conflicts, and which had conflicts, that was kind of neat. Though they used a terrible SCM, IBM clearcase or something, that was less neat :)
< morcos>
gmaxwell: can we go through exactly how you think the grace periods should work... i never looked at manual prune before but looks like it takes a block or a height
< morcos>
sorry time
< fanquake>
wumpus ah nice.
< fanquake>
wumpus I was thinking if you could have a "projects" like screen, in which you could stack PRs, and see what breaks what.
< morcos>
gmaxwell: so you think if you give it a time it should automatically subtract 7200 from it and then call findEarliestAtLeast?
< morcos>
what happens if you give it a block (no grace period?). i think that seems reasonable'ish. but it would be counterintuitive if it didn't do exactly what you expect with a block i think
< wumpus>
no stupid mistakes? I'm as surprised as you are :)
< fanquake>
Maybe I didn't look closely enough
< wumpus>
hah the hardlinking trick works, both for block files and ldb files, created this script to do it: https://gist.github.com/laanwj/3c4614a23e072cbb3d39090da1834a68 . Creates a "copy" of the whole state almost instantly. I've tried various things (including pruning) and the original state remains unaffected.
< luke-jr>
wumpus: but fCheckForPruning is set in manual pruning mode after a new block is connected?
< wumpus>
luke-jr: it shouldn't be
< wumpus>
manual pruning is manual pruning, the check should never be enabled
< wumpus>
if it is, that is a bug
< luke-jr>
FindBlockPos/FindUndoPos?
< luke-jr>
or is fPruneMode false for manual pruning?
< wumpus>
no,that should be true if any pruning is to be done
< wumpus>
in retrospect it'd have been better to make fPruneMode an enum, which can be off, manual and auto
< wumpus>
the boolean gymnastics with fCheckForPruning is kind of difficult to follow
< wumpus>
I assumed it would be enabled when auto pruning is enabled, but apparently it's also enabled in a few other places
< wumpus>
luke-jr: ah I got understand it now - nPruneTarget is 0 in manual pruning mode
< wumpus>
luke-jr: so FindFilesToPrune does nothing in manual pruning mode
< wumpus>
even though it may get called (even automatically in some cases), nothing will happen.
< * sipa>
suggests adding comments to clarify these things
< luke-jr>
aha!
< wumpus>
I wish it'd simply pass parameters instead of signalling using global flags
< gmaxwell>
well, or change to an enum.
< wumpus>
or both. I mean the global enum is fine for the mode but it shouldn't change over time
< luke-jr>
these changes sound even more invasive than that PR :P
< wumpus>
depending on what function ran last. This is impossible to follow. Anyhow I understand marcofalke's rationale for #9524 now, it's very easy to break that code and belt and suspenders wouldn't hurt. Except that even with #9524, it will get into FindFilesToPrune in some cases, so if that check is broken, bad things will happen *automatically* too
< wumpus>
jonasschnelli: well it's a magic option value but it's only for the option, it doesn't get represented internally as 1
< luke-jr>
tbh, I've had second thoughts about the manual pruning design (seems it'd be nicer to do "automatic pruning always, but with named barriers that must confirm being done up to <height> before pruning it"), but that's beyond this topic
< wumpus>
jonasschnelli: I think the reason or choosing 1 was that -prune will work
< wumpus>
luke-jr: I like the current manual pruning from an API point of view
< luke-jr>
wumpus: yes, but it doesn't scale well if you have 2 external apps using the node
< cfields>
hi
< wumpus>
luke-jr: it's very simple. Automatic pruning is disabled and the client/admin can choose what to prune. Also useful for testing the pruning stuff without too much complexity
< wumpus>
luke-jr: the implementation is convoluted though
< morcos>
wumpus: i think two more are needed for 0.14, both very small #9760 and #9761 (9761 is already included in one of the marked ones)
< paveljanik>
proposed topic: release status, where can we help...
< gribble>
https://github.com/bitcoin/bitcoin/issues/9761 | Use 2 hour grace period for key timestamps in importmulti rescans by ryanofsky · Pull Request #9761 · bitcoin/bitcoin · GitHub
< wumpus>
luke-jr: but I'm happy someone implemented it anyhow
< * luke-jr>
nods
< kanzure>
hi.
< petertodd>
wumpus: same: my OpenTimestamps servers actually need this feature
< wumpus>
release status: running as hard as we can but staying in the same place
< sipa>
i think we're very close.
< morcos>
wumpus: boo! we're not staying in the same place
< gmaxwell>
I am becoming increasingly happy with the release.
< sipa>
let's be optimistic: everything we're fixing is an improvememt
< gmaxwell>
Two weeks ago I was chewing my nails feeling like we were at risk of shipping something that wouldn't meet our standards of quality, and now I am not worried. :)
< jonasschnelli>
hah. Good.
< wumpus>
but I think we need to decide when we can do release candidate 1, which is a test release anyway, when rc1 is out it's bound to find more issues
< sipa>
i think we can do rc1 today?
< gmaxwell>
I would be fine doing it _now_, now. There are AFAIK not anything in the pipe which are disruptive enough issues that they'd degrade our rc1 learning, though a rc2 will be guarenteed.
< paveljanik>
rc1 for the weekend is fine!
< sipa>
or at least branch off today
< luke-jr>
I don't think we have any critical problems blocking a rc1
< jtimon>
sipa: why branch of if we can't rc1 ?
< wumpus>
I don't think so either
< cfields>
no more blockers from me either
< luke-jr>
jtimon: to begin merging for 0.15?
< wumpus>
right, because more and more pulls for 0.15 are waiting
< * jtimon>
nods
< jonasschnelli>
IMO the two import multi fixes are not critical and can go in after rc1 (or even in 0.15 in the worst case).
< gmaxwell>
all you naughty people not banging away at getting 0.14 ready to go.
< morcos>
jonasschnelli: well lets decide that
< gmaxwell>
jonasschnelli: I think 0.15 would be really unfortunate since they're interface changes that users may have to accomidate in their software. (and also are covering corner cases that could result in funds losses)
< wumpus>
I think the fixes are all important, and should get either into 0.14.0 or backported to the 0.14 branch if they don't, but not everything should be regarded as yet another thing to hold up rc1
< morcos>
yesterday people were saying it was bad to release with something that coudl silently not find funds
< gmaxwell>
but no reason to hold rc1 for them. They're well understood.
< jonasschnelli>
Yes. I think the should be in 0.14. Just worst case if we spun of rc1 and chaincode-labs colabses. :)
< wumpus>
anyhow tomorrow sounds good to me, let's try to get in what we can get in
< gmaxwell>
it's not like someone is going to encounter one of them running rc1 and leave us going "shit was that the know issue or something else?"
< luke-jr>
#9619 is a fix, but not really critical since anyone affected needs a workaround in 0.13 anyway (just pushed an amend-with-no-changes because the Travis error seems impossible)
< gribble>
https://github.com/bitcoin/bitcoin/issues/9619 | Bugfix: RPC/Mining: GBT should return 1 MB sizelimit before segwit activates by luke-jr · Pull Request #9619 · bitcoin/bitcoin · GitHub
< achow101>
what's the point of making an rc1 if we're going to need rc2 anyways?
< sipa>
achow101: exposure
< gmaxwell>
achow101: to start getting people using it.
< wumpus>
achow101: get the code actually tested?
< gmaxwell>
more*
< wumpus>
what is the point of doing rc1 at all if not?
< kanzure>
and seeking bug reports
< jonasschnelli>
I think we should plan enough time to fix stuff that gets report after rc1...
< sipa>
jonasschnelli: the release schedule already has 2 weeks left
< wumpus>
achow101: I don't think it ever happened that a major release didn'tneed a rc2
< luke-jr>
2 weeks can go fast
< jonasschnelli>
Other can work on fixes reported in rc1.
< gmaxwell>
achow101: what we generally don't want to do is to ship an RC1 with issues bad enough that it will harm the testers seriously, or which will fail in mysterious ways that we can't learn from. E.g. if we had a known crash fix, we would hold rc1, so that we wouldn't worry that every user crash report might have been an unknown issue.
< jonasschnelli>
*Others
< morcos>
my only concern is that if we do rc1 everyone will be distracted with that and not this last few remaining PR's.. but i guess i don't really care either way
< cfields>
wumpus: just forget the bump to v0.14 again and thereby guarantee an rc2 :p
< wumpus>
cfields: lol exactly
< morcos>
seems like if someone would just review those PR's we might be able to get them merged today
< gmaxwell>
well I think an RC2 is already guarenteed if we do an rc1 ASAP. But thats fine. Much better than not doing the rc1.
< achow101>
ok
< wumpus>
a RC2 is guaranteed in any case, not just if we do rc1 asap
< wumpus>
that's my point, we don't know of all the issues yet
< morcos>
almost all the complication is in new tests.. reviewing the code changes is pretty simple
< wumpus>
the only one I doubt about is #9773, at it is still WIP
< gribble>
https://github.com/bitcoin/bitcoin/issues/9773 | WIP: Return errors from importmulti if complete rescans are not successful (on top of #9761) by ryanofsky · Pull Request #9773 · bitcoin/bitcoin · GitHub
< luke-jr>
it's a new feature, so we could just say "don't use this without being aware of the risks"
< paveljanik>
mark it as experimental then?
< sipa>
paveljanik: meh, i think we're close enough to not do that
< luke-jr>
I mean in rc1 only
< wumpus>
well even with that change it can be marked as experimental
< wumpus>
it is new code afterall
< gmaxwell>
I'm not too worried about it in rc1.
< wumpus>
there are probably still problems with it that we haven't found, which will be found by people testing it
< paveljanik>
we can mark it as experimental in the release notes only...
< wumpus>
so yes, experimental makes sense. Though a new major release should be considered experimental entirely
< gmaxwell>
people running rcs are the least likely to lose funds due to a rescan limitation, and there won't be many of them.
< morcos>
i think it is a mistake to call it experimental
< morcos>
we don't want to devalue the meaning of that word
< wumpus>
ok...
< morcos>
sometimes we may want to have things that are actually experimental and we don't want people to think we just always say that
< wumpus>
"this feature is experimental level 4"
< sipa>
haha
< kanzure>
nasa technical readiness levels
< CodeShark>
The whole thing is experimental :p
< morcos>
this is known to the state of CA to be experimental
< petertodd>
morcos: "dangerously experimental"
< petertodd>
morcos: lol
< gmaxwell>
Yea, thats why I was not supporting expiremental.
< instagibbs>
morcos, accounts... deprecated!
< morcos>
what.. is that an announcement? a question?
< morcos>
i hate accounts
< gmaxwell>
For rc1 we can simply say that there are in-flight improvements to that feature link to the PRs. if we really want to... in a reply to the announcement.
< sipa>
let's just review the outstanding patches, they are tiny
< instagibbs>
morcos, we use "deprecated" in things that are lasting forever in practice, bad joke
< jtimon>
mhm, why 9619 doesn't go in for 0.14?
< morcos>
oh.. yeah
< sipa>
with some luck we can get everything merged
< sipa>
branch and rc1 tomorrow
< wumpus>
gmaxwell: yes, that there is still work in progress, that's better and more clear than the word experimental
< sipa>
with whatever is in
< morcos>
ok, so can we just pick a time. wumpus is going to call rc1 tomorrow morning.. it tonight we can convince people to merge a few of these other things... that'll leave less for rc2
< morcos>
that was "if tonight" , if not , then ok
< sipa>
wumpus: is that fine by you?
< wumpus>
yes, that's fine with me
< wumpus>
I'm okay with another day, let's just not let it slip another week, that next thursday we're again wondering when to do the rc1 :)
< sipa>
next thursday we should be talking about doing rc2
< wumpus>
yes, ideally
< gmaxwell>
We're at a point where beyond the couple in flight PRs little to no more improvement is happening, which is when we need more input.
< achow101>
so rc1 tomorrow regardless of whether those three prs get merged?
< sipa>
achow101: that's what i suggest, yes
< paveljanik>
+1
< achow101>
^
< sipa>
are we ok with talking about things beyond 0.14?
< wumpus>
sure!
< luke-jr>
new #topic?
< sipa>
i'd briefly like to talk about randomness
< wumpus>
#topic randomness
< luke-jr>
that's random.
< sipa>
we currently have 3 "levels" of randomnesz
< wumpus>
we need a random number of levels of randomness
< wumpus>
yes.
< wumpus>
why are there three?
< instagibbs>
can you explain "getsecurerandbytes"
< instagibbs>
im going through code now but..
< wumpus>
I mean we need one for wallet keys, I understand that
< sipa>
the last one is used for privaye keys
< jtimon>
why 9619 doesn't go in for 0.14?
< jonasschnelli>
jtimon: wrong topic
< sipa>
it's as secure as getrandbytes if all goes well, but it's more paranoid
< sipa>
so, i've been playing with a chacha2p based rng instead of fastrandomcontext
< jtimon>
jonasschnelli: suggested topic then
< wumpus>
I think I'm fine with an extra paranoid level for wallet keys
< sipa>
*chacha20
< sipa>
which takes around 10ns for a 32-bit rand
< wumpus>
it's much more important to have good randomness there then in almost any other place in computing currently
< instagibbs>
"GetStrongRandBytes" <--- this it?
< rabidus>
return 4
< sipa>
the current fastrandomcontext takes 1.5ns, but with extra optimizations it can be made comparablr
< sipa>
and if we have that, i think we can use strong randomnes for everything
< wumpus>
for non wallet keys we can probably do with one level
< wumpus>
is that your plan sipa?
< sipa>
basically i suggest making all RNGs cryptographic
< sipa>
but the fast one is not thread-safe, doesn't reseed, doesn't protect against VM reloads
< wumpus>
so merge the fastrandomcontext and somewhatmoresecure level, and keeping that and the ultra-paranoid one for wallet keys
< sipa>
i realize the "only two randomness levels" is a bit of an abstract goal
< wumpus>
yes the fastrandomcontext is supposed to only eb used from one thread at a time
< morcos>
are there any inner loops that use random numbers?
< petertodd>
wumpus: the ultra paranoid one can also use the chacha code
< wumpus>
because it's for inner loops
< sipa>
yes, the wallet coin selection
< sipa>
i benchmarked it, making it chacha20 doesn't affect its performance
< wumpus>
any locking for synchronization there would be pretty bad
< wumpus>
there's also an inner random loop in the address selection code IIRC
< jonasschnelli>
Do we need cPRNG for coin selection?
< sipa>
yup
< wumpus>
sipa: yes chacha20 is fast, but the problem is the thread safety
< sipa>
jonasschnelli: no, we don't
< gmaxwell>
sipa wants to reduce the codebase complexity.
< sipa>
but chacha20 is so fast i don't care
< wumpus>
if you want to be able to share a random context between threads, you end up with lots of synchronization overhead
< jonasschnelli>
Yes. This would be good.
< wumpus>
that's why the inner loops use a non-threadsafe random context, =which doesn't matter as it's only used by one thread
< bitcoin-git>
[bitcoin] gmaxwell opened pull request #9779: Update nMinimumChainWork and defaultAssumeValid. (master...update_chainparams) https://github.com/bitcoin/bitcoin/pull/9779
< sipa>
having clearer expectations about the rngs may simplify getting rid of openssl later
< wumpus>
the thread sync overhead is where the overhead would be
< sipa>
though that's not something to discuss now, i think
< wumpus>
so how would you cope with that sipa? or would it all be non-shared context?
< sipa>
so 1) replace fastrandomcontext with a bit more featureful chacha20 based one
< wumpus>
yes, depending on openssl for just randomness is fine, there's no urgent reason to get rid of that, and it seems controversial for some people
< sipa>
2) see where the current non-strong-getrandbytes can be replaced with fastrandcontexts, and replace everything with getstrongranbytes
< wumpus>
that makes sense
< gmaxwell>
one of the harder points (beyond threading) is that we need to manage which RNGs are robust against a VM image being snapshotted and restored. Coinselection using the same randomness again after a VM restore would be harmless. Choosing a crypto nonce would be much less harmless. If a RNG needs to be outputing data intially after restore there is unfortunately a runtime cost (esp on ARM).
< wumpus>
also I think we need an abstraction for 'kernel randomness', this came up recently in context of OpenBSD, which doesn't have /dev/random/urandom available in all contexts
< gmaxwell>
Pieter and I have been debating this a little bit the past could days.
< wumpus>
the modern way,sandbox-proof way seems to be to use a specific system call fo that, but it differs per OS unfortunately
< sipa>
wumpus: yup, that would go into the (singular) getstrongrandbytes implementation
< cfields>
isn't there an old PR for exactly that abstraction?
< gmaxwell>
wumpus: we do have a function for OS randomness, it should be smarter of course... but it was only fairly recently introduced.
< Victorsueca>
maybe you could get rid of getrandbytes? so we keep the fast one for simple operations and the secure and paranoid one for important stuff and you can focus on implementing more features on those two
< gmaxwell>
GetOSRand()
< wumpus>
(linux also has a "getrandom" system call that can be used instead of /dev/urandom, in sandboxes)
< sipa>
Victorsueca: read the above discussion, that is exactly what i am proposing
< wumpus>
gmaxwell: ok
< gmaxwell>
wumpus: yes, in newer systems. We do mention using that in the PR that implemented GetOSRand I think. (getentropy)
< wumpus>
gmaxwell: in any case, that is the lowerst level, it shouldn't be regarded as 'a level of randomness'
< gmaxwell>
so I think we know what we need to go there.
< wumpus>
indeed, getentropy is bsd, getrandom is linux
< Victorsueca>
sipa: ohh, I somehow misread that you where proposing to merge the fast and the middle one to make it simpler
< sipa>
FWIW linux 4.8 also switched to chacha20 for /dev/urandom
< wumpus>
yes
< gmaxwell>
The framework I think about this is that we have several randomized algorithims where there are basically no cryptographic guarentees needed... coin selection, addr man buckets, and tests being some examples. These need to be fast but can all use just a local context, need no resistance against reversal (somsone steals your ram and recovers older randomness) or prediction (vm saves repeat rando
< petertodd>
gmaxwell: re: VM image being snapshotted and restored, that's basically a case where reusing a secret is by itself harmful - is there an example in Bitcoin where that's the case, now that we do deterministic signing?
< gmaxwell>
mness).
< gmaxwell>
So thats one set of uses.
< sipa>
Victorsueca: i'm proposing to make the fast one stronger (but not much slower), and then things using the middle one need to be judged on a case by case basis whether they can use the new fast one, or the strong one
< sipa>
Victorsueca: after that, the middle one goes away
< gmaxwell>
Then we have other uses where we have randomized behavior which does have stronger security requirements, like ping nonces which need to be strongly unforgable to prevent peers hiding their latency.
< Victorsueca>
sipa: makes sense
< gmaxwell>
But even if they're broken it just turns into DOS attacks.
< wumpus>
it's strong, but far from as strong a requirement as wallet keys
< gmaxwell>
Then we have things like long term keys which we do infrequently and basically no cost is too high. And they have to meet basically every security characteristic we can imagine.
< wumpus>
indeed
< cfields>
suggested similar next topic: clocks
< gmaxwell>
the second class often has to be moderately fast too.
< gmaxwell>
Pieter would like to collapse this class hierarchy some by making the first class use the second while making the second fast enough that it's fine to do so.
< Victorsueca>
cfields: clocks as in CPU clock or as in timestamp?
< wumpus>
Victorsueca: please wait until the topic is actaully started
< Victorsueca>
ohh, sorry
< wumpus>
though I think we've wrapped up randomness
< sipa>
agree
< wumpus>
#topic clocks
< gmaxwell>
I have a little doubt that this is possible, because I think the second may need to deal with reversal resistace and prediction resistance, which cannot be done for free. (e.g. you must mix in TSC and/or RDRAND at every use.)
< instagibbs>
he has to gavel before we can switch
< gmaxwell>
okay!
< wumpus>
gmaxwell: oh, didn't know you were still typing, sorry
< cfields>
I have some local changes that implement the concept of different clocks/time_points/durations. The objective is for them to be incompatible with each-other.
< gmaxwell>
thats fine! just some things to think about.
< wumpus>
cfields: yes, that seems the way to go about it
< sipa>
cfields: f i may interject... i was thinking about creating a generic int class wrapper that supports no implicit conversioms
< gmaxwell>
cfields: pieter and I were talking about type safty recently, and pieter suggested a scheme for introducing more integer types which will never implicitly be converted.
< wumpus>
most importantly we should start using monotonic timestamps in the network code where possible
< cfields>
that sounds fine, but i'm not sure that they're entirely related here. The (my) objective is to stop storing time as an int, and instead as a time_value
< sipa>
cfields: or are timestamps in a c++11 world not something that fit in integers?
< cfields>
that way it can be represented in sec/msec/whatever whenever it's needed
< cfields>
sipa: exactly
< cfields>
it also enforces timestamps that can't be used on the wrong clock
< sipa>
i'm confused
< gmaxwell>
I have suggested in the past that we consider constructing a monotonic local clock, but wumpus seemed to not like the idea. which I think is orthorgonal to the type safty thing, but it would perhaps make time more sane in the codebase.
< wumpus>
cfields: in general that sounds good, though in some structures such as the block index we want to use as compact types as possible
< gmaxwell>
saftey*
< gmaxwell>
doh
< cfields>
wumpus: sure, you can always get an int out of it if you want
< wumpus>
gmaxwell: huh I'm all for using monotonic clocks were possible, they're just not good for everything
< gmaxwell>
safety**
< sipa>
cfields: my point was to have a template<typename tag> class non_convertible_int
< cfields>
also, i believe the class is actually not bigger than an int. It's just not convertable to int
< wumpus>
cfields: well if it represents micro/nanosecond it needs to be at least uint64 :)
< sipa>
and then have typedef non_comvertible_int<systemtime> systemtime_type
< sipa>
and systemtype_type is what is used
< gmaxwell>
you would get_int() on the object to get an int. You would just get a conversion by surprise.
< sipa>
*systemtime_type
< gmaxwell>
I would _hope_ we can construct something that at runtime is _exactly_ equal to using an integer.
< gmaxwell>
I think we should do this far more broadly than just timestamps however.
< sipa>
we can't use that in blocks, etc
< wumpus>
indeed - block times /consensus are a special case
< cfields>
sipa: we don't use timeval in blocks either though, we convert from the clock
< sipa>
but perhaps we should use those types in network state, measuring speed, ...
< cfields>
sipa: i'll demonstrate with code, probably easier that way
< wumpus>
right
< sipa>
cfields: what i want to address is the fact that an int or int64 can now mean microseconds, milliseconds, or seconds, and either system time, or monotonous time, or network-adjusted timr
< sipa>
it's fine that those are int-like, but they shouldn't be convertible from one into another
< cfields>
sipa: and that's exactly what i've addressed. Each of those gets its own type, and they're not convertable to eachother. But you can do a duration_cast<std::chrono::seconds>(foo) and get an int64_t seconds value out
< sipa>
anyway, for everything but consensus data structures, perhaps there are more c++ish ways
< sipa>
cfields: let's discuss this outside of theeeting
< gmaxwell>
This problem exists far beyond timestamps however. Use a node ID as a tx count? no problem. Use a vin index as a block number? no problem. Use a bytes sent as a relay bool? no problem.
< gmaxwell>
We have multiple times had potentially serious bugs from the general issue of implicit conversions.
< cfields>
gmaxwell: yes, agreed that the tag would be very useful
< gmaxwell>
(or in the case of sighash single, an actual consensus behavior flaw)
< cfields>
sipa: np
< sipa>
jtimon had a topic as well
< wumpus>
using enumerations instead of booleans would also go a long way
< jtimon>
well, just a question really
< jonasschnelli>
I also have a little proposal
< gmaxwell>
all the data is there in the compile to prevent these mistakes, we're just not exposing it right. :)
< jtimon>
I guess it can wait after the meeting
< wumpus>
especially for functions that have tons of boolean arguments in succession, that's just crazy
< sipa>
wumpus: yeah, generally useful
< gmaxwell>
wumpus: yea, using bools, also using structs to get named parameters... lots of things we can do.
< cfields>
wumpus: for sure
< jtimon>
or answered fast enough that it doesn't need its own topic: "why 9619 doesn't go in for 0.14?"
< gmaxwell>
true false true true false die die die. ... I hate counting aruments when changing things.
< morcos>
jtimon: i think because there was an error reported and no explanation that it was fixed or wasn't really a problem. that's at least why i haven't looked at the PR.
< wumpus>
(well with some IDEs you can see what gets assigned to what parameter because it parses the interface, but that's not something you can realy on that everyone has available)
< wumpus>
gmaxwell: exactly
< wumpus>
jtimon: what was your topic?
< instagibbs>
gmaxwell, the fun really starts when you drop an arg with a default value at the end
< cfields>
(or three)
< jtimon>
morcos: that explains why is not merged, not why it's not labeled 0.14, but ok I guess
< wumpus>
jtimon: let's reverse the question: why would it need to be added for 0.14?
< jtimon>
wumpus: including 9619 in 0.14
< jtimon>
it's a bugfix and is simple enough, why not?
< sipa>
i think it can go in, it's trivial and very small in scope
< gmaxwell>
basically without it a plausable downstream gbt user that changes the transaction set could construct an overlarge block, is that the concern there?
< wumpus>
I'm fine with merging it before 0.14, if its sufficiently reviewed and teste
< wumpus>
it will however not hold up rc1
< gmaxwell>
it looks trivial and small in scope, and if my understanding is correct it should go in.. but sure, nothing should hold up rc1.
< gmaxwell>
at least nothing that we know of now.
< sipa>
fair
< jonasschnelli>
Not sure if this makes sense, But I propose to rename the ./qa dir to ./test (or something that sorts after ./src). I think it would be better to have the RPC tests further down in the PRs diff.
< jtimon>
sure, was simply surprised it didn't had the 0.14 label since it's not really that new
< Victorsueca>
I see lots of utACKs but not any ACK, so trivial it doesn't need testing I assume?
< gmaxwell>
short of some kind of crash eat money doom bug showing up before we manage to get it out (and even that shouldn't delay _constructing_ RC1; if doom shows up we can abort launch)
< wumpus>
gmaxwell: sure, there are always serious enough problems possible that should postpone the release
< gmaxwell>
jonasschnelli: On that I'd like to get into a state where make check is running those tests. They are most of our tests now, and the most valuable.. and really people building with compilers we've never seen before _REALLY_ ought to be running them.
< wumpus>
jonasschnelli: don't really have an opinion on that, does it matter much how things are sorted?
< gmaxwell>
Why would the sorting matter?
< jonasschnelli>
As long as review is a bottleneck I think it matters
< jonasschnelli>
But maybe I'm alone with that opinion.
< gmaxwell>
OH so they show up lower on github.
< sipa>
/zzztest
< wumpus>
my biggest pet peeve is that they're called RPC tests, not 'functional tests' or such, they're testing much more than RPC these days :)
< jonasschnelli>
./test_functional
< gmaxwell>
It is sometimes a bit confusing that the tests show up first... otoh it can be informative to read the test before the change in the cases where the test is especially good. :)
< wumpus>
but not serious enough to actually go on renamind directories
< sipa>
also, pull-tester should be merged in rpc-tests
< gmaxwell>
wumpus: they are "system tests" rpc is incidental.
< sipa>
we don't have pulltester anymore, and it's annoying to always change directory :)
< wumpus>
gmaxwell: yes, it doesn't matter what interface they use, whether it's RPC or P2P or ZMQ or REST or command line arguments etc
< instagibbs>
we're over time
< instagibbs>
btw
< jonasschnelli>
Okay. I'll do a cleanup PR once we branch off.
< wumpus>
instagibbs: ah yes
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Feb 16 20:01:55 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< jnewbery>
wumpus: Agree, and can we name it something other than pull-tester?
< Chris_Stewart_5>
sipa: gmaxwell So if I understand you guys, you want to create custom types for all time related structures inside of core?
< gmaxwell>
petertodd: nice ACK on 9779.
< Chris_Stewart_5>
even further, integer related structures?
< sipa>
Chris_Stewart_5: well it seems c++11 has things like that already, and they're more featureful than just integers
< cfields>
Chris_Stewart_5: i'm suggesting that we switch to std::chrono::duration and std::chrono::time_point
< wumpus>
jnewbery: yes, pulltester is a weird name now, it's simply a launch framework for the tests
< Victorsueca>
midnightmagic: "and really people building with compilers we've never seen before _REALLY_ ought to be running them." <--- I tried it once on windows but it looks like they're broken and only work on linux
< sipa>
Chris_Stewart_5: but yes, making ints not convertible into other int-like-types can fix bugs
< jnewbery>
test-launcher.py or test-runner.py seem fine
< gmaxwell>
wumpus: re 9779, assuming that gets ACKs feel free to merge that along with the version bump. (the assumevalid was 6119 blocks behind.)
< wumpus>
Victorsueca: that's simply not true! they work on at least openbsd, freebsd, linux, and windows (in wine at least)
< Chris_Stewart_5>
sipa: Sure, no arguments with that. Just wasn't sure if that was _exactly_ what you guys were saying in the meeting.
< wumpus>
Victorsueca: oh and OSX
< Victorsueca>
wumpus: hmmm should try it again
< Chris_Stewart_5>
sipa: But conversions will still be able to be done right? Just with explicit function calls instead of casting correct?
< wumpus>
Victorsueca: our tests are very well maintained on many platforms
< sipa>
Chris_Stewart_5: of course
< sipa>
otherwise it's useless
< cfields>
Chris_Stewart_5: only to a degree. You shouldn't be able to compare a system_clock time_point to a steady_clock one. Because that makes no sense.
< gmaxwell>
They tests have extra dependencies so its quite plausable that on some systems that compile the code the tests fail.
< wumpus>
cfields: or a duration to a absolute time value!
< cfields>
wumpus: right
< Victorsueca>
wumpus: which one should I be running to run all tests?
< Victorsueca>
ahh I think I found it
< wumpus>
gmaxwell: it should disable the tests in question in that case, e.g. if there's no zmq if doesn't run that test
< jtimon>
wumpus: right, functions with many bools could use thier own flags or something perhaps (sorry, was catching up on the previous discussion)
< jonasschnelli>
Victorsueca: mind the -win parameter
< wumpus>
jtimon: yes, either bitfield or individual enums, everything is better than bool,bool,bool,bool
< cfields>
so instead, we get the concept of a clock, which gives you time_point's with some meaning. a time_point, which is an absolute time (but not necessarily wall-time) on some clock, and a duration, which is the difference between 2 time_points
< sipa>
wumpus: for cases where it's excessive you can emulate named arguments using a struct
< jnewbery>
jonasschnelli: My preference would be to also move the bitcoin-util-test.py and bctest.py (and data for those tests) from src/test to whatever you rename qa to. Those tests seem much more like the integration tests in qa than the unit tests in src/test
< jtimon>
and I really hate we have so many bools with default values
< sipa>
wumpus: for example a struct ATMPArgs that takes in its constructor all 'required' fields for ATMP, and has methods to change optionals
< Chris_Stewart_5>
cfields: Wouldn't that comparison have to be enforced through code review and not a compile time failure? Sorry not super familar with the data structures..
< jonasschnelli>
jnewbery: Indeed. I'll try to move them.
< gmaxwell>
I believe a struct for inputs can be nearly zero runtime overhead for functions with many arguments.
< sipa>
Chris_Stewart_5: it will result in compile time failure; a timepoint for a steady clock would be a different data type
< wumpus>
sipa: c99 designated struct initializers were always nice to use for emulating named arguments in C, too bad that never made it to C++
< cfields>
Chris_Stewart_5: no, it's enforced by the compiler. A time_point knows what clock it belongs to
< sipa>
So you'd call AcceptToMemoryPool(ATMPArgs{std::move(tx)}.max_fee(fee))
< gmaxwell>
Chris_Stewart_5: enforcing things through code review that could be enforced by the machine is a losing strategy. Code review is essential but it should be focused on the bugs the machine cannot prevent.
< petertodd>
gmaxwell: heh, yeah, I wanted to make a point of what the trust model was, particularly since -assumevalid was (partly) my idea! :)
< wumpus>
sipa: in any case, we also shouldn't go over the top with this, trying to emulate something in c++ that it is not :)
< jonasschnelli>
Victorsueca: python3?
< Chris_Stewart_5>
gmaxwell: No doubt. I'm all for type safety. Compilers are our friends :-)
< jonasschnelli>
(requires p3)
< sipa>
Victorsueca: looks like you're trying to run with python2
< Victorsueca>
ahh forgot python stands for 2 and py stands for 3-2 hub
< sipa>
Victorsueca: no, the first line of the file states the interpreter to use, but i guess that only works in unix-like systems
< wumpus>
you should run all the bitcoin core scripts with python 3
< jtimon>
gmaxwell: ack on adding the rpc tests on make check
< Victorsueca>
here on windows if I run py it auto-detects whether 2 or 3 should be used
< Victorsueca>
it's just i'm used to write python
< gmaxwell>
wumpus: MISRA C (standard for safety critical C code targeting embedded systems) requires that no generic types be used, and certantly no sizeless generic types. And that is in C where you still get the implicit conversion and can't really avoid it, but the standard argues that static analysis tools can still catch your errors so long as you've exposed the type information. Libsecp256k1 is pre
< gmaxwell>
tty close to this now.
< wumpus>
well if the first line contains python3 it should be run with python 3, lacking that there is no general way to detect whether code is python2 or python3. But believe me, the bitcoin core python scripts need to be run with python 3.
< wumpus>
gmaxwell: the most significant non-MISRA thing about secp256k1 is probably that it allocates a context on the heap?
< bitcoin-git>
[bitcoin] jnewbery opened pull request #9780: Suppress noisy output from qa tests in Travis (master...travislogging) https://github.com/bitcoin/bitcoin/pull/9780
< gmaxwell>
wumpus: Well MISRA doesn't actually forbid anything but requires that for every violation you write an exception document. So the context is one where the exception document is very easy to right: this allocation is pure initilization, not runtime.
< gmaxwell>
What I've found harder to deal with is that it requires no function have multiple returns. As there are some cases where we do where fixing it feels like it would objectively worsen the code. But their argument is also pretty compelling
< wumpus>
yes, there's something to be said for that, especially with regard to cleanup and such
< gmaxwell>
It would also be very easy for us to make it possible to make libsecp256k1 mallocless with ifdefs, it's specifically structured to enable that.
< wumpus>
though in some cases it can lead to terribly verbose code when combined with error handling
< wumpus>
e.g. either deeply nested if() or some status flag that is checked every other statement
< cfields>
wumpus: but those easily fixed with a goto ;)
< gmaxwell>
(My goal though not a priority is to make libsecp256k1 MISRA conformant eventually)
< gmaxwell>
cfields: forbidden by MISRA C. (though like anything else you can make exceptions)
< jtimon>
jonasschnelli: regarding your rename, if rpc-tests/ and pull-tester/ in qa (as I think sipa suggests ), it may make sense to take the opportunity to rename qa as well
< wumpus>
cfields: hah! I didn't expect those to be allowed if multiple returns are not
< cfields>
gmaxwell: was just joking about trading one thing for another
< gmaxwell>
Gotos that go backwards also forbidden by an extra rule, so at least error handling doesn't hit those.
< wumpus>
cfields: I always tend to use goto's in C code for cleanups, kernel style
< gmaxwell>
cfields: well I wasn't goto is actually an excellent way to handle error handling in C in some cases. It's very similar to multiple returns in its downsides, and yet few people care about peppering functions with 15 returns.
< cfields>
wumpus: yes, they seem pretty necessary after using raii for long enough
< gmaxwell>
The MISRA document is a really good read FWIW. It's pretty thoughtful in its recommendations.
< gmaxwell>
though not so applicable to C++. :)
< cfields>
gmaxwell: sure, agreed. I just used it as an example because it's often multiple-returns and gotos blindly spouted as "thou-shalt-not"s in c-code
< wumpus>
I can't come up with a good reason to use backwards gotos though
< cfields>
gmaxwell: well, no-multiple-returns carries some weight in c++ too as they inhibit rvo. Which can be very useful for structures that you assume move freely/cheaply
< Victorsueca>
ok... now this is a mess.... I use WSL to cross-compile windows binaries, now all the tests have UNIX-like paths and can't run them on windows
< isle2983>
gmaxwell: do you try any clang/LLVM plugins for checking MISRA? I am just wondering if it is worthwhile exploring
< wumpus>
gcc computed goto is even more evil: arrays of labels, though it's used in e.g. the python bytecode dispatch loop for the last bits of speed optimization
< gmaxwell>
isle2983: not clang/llvm ones, but I've used other static analysis tools that can check it, in particular pc-lint... They're of some value.
< wumpus>
Victorsueca: strange - python code doesn't get compiled as such, it must be one of the generated py files that has a path in it. You could try explicitly setting the environment variable BITCOIND to the path of bitcoind, that will override the built-in guess
< gmaxwell>
wumpus: some kinds of valid but complex flow control is just not expressable in C without a loss of efficiency. So you could imagine some kind of hyper optimized inner loop that uses backwards goto. I don't recall any case where I've been forced into that, though I have run into cases where performance required do{}while() and non-trivial code inside for expressions, and forwards goto out of a
< gmaxwell>
loop that also had breaks and continues.
< wumpus>
gmaxwell: yes, forwards goto out of a loop makes a lot of sense, if you want to break out of multiple levels
< gmaxwell>
I wish C(++) had named continue/break.
< wumpus>
gmaxwell: and these things matter, I mean if efficiency is not important you wouldn't be using c in the first place
< gmaxwell>
(Rust has named breaks)
< gmaxwell>
(also more important because it lacks the other options C has for that)
< wumpus>
yes I remember quite some languages had them ,though I can't think of any right now
< wumpus>
didn't know that rust did
< sipa>
in c++ it's more complicated because you can't jump over variable initializations
< cfields>
gmaxwell: i suspect that architects assumed throw/catch would be used for breaking out of multiple loops. But that's seen as poisonous to most, i think
< wumpus>
ah, java has them apparently
< gmaxwell>
cfields: youch. :P
< cfields>
gmaxwell: my thought exactly :)
< wumpus>
cfields: exceptions as a control flow statement? speak about shooting a fly with a thermonuclear missile :p
< gmaxwell>
beyond the other issues with exception, they're slow... sooo.
< wumpus>
right, they're slow exactly because they don't know where they're going to be catched, so they're a bad fit if you know exactly where you wantcontrol flow to go
< wumpus>
the code for exceptions and stack unrolling is extremely complex, I looked at it some time there's even some cute VM involved that interprets DWARF debug info
< cfields>
wumpus: heh, i once worked on some code that used it to parse some data, construct the right handler, and throw it. Then the catch was an abstract parent, and it would continue on with the newly-constructed handler
< wumpus>
cfields: hahaha
< cfields>
it was ugly, but surprisingly effective :)
< wumpus>
cfields: anti-reverse-engineering?
< cfields>
wumpus: heh, it would seem. no idea what the historical reasoning was
< gmaxwell>
later you find out the guy that wrote it didn't know that function pointers or virtual methods were possible. :P
< wumpus>
like anytiing so evil it sounds like a hell to debug
< cfields>
gmaxwell: entirely possible :)
< wumpus>
it reminds me of that saying that you should always expect that the person maintaining the code after you is a crazy axe murderer that knows where you live, this certainly classifies as inhuman cruelty to your successors
< jeremyrubin>
sorry for misisng meeting! I was off by an hour :p
< jeremyrubin>
I think w.r.t. entropy levels we should keep 3
< jeremyrubin>
deterministic is really useful for writing tests/benchmarks IMO
< wumpus>
it's only useful for the tests and benchmarks, and could be in test_util.cpp
< jeremyrubin>
That's fine to me!
< cfields>
jeremyrubin: we have siphasher, which we use for deterministic randomness, and i don't think it really needs to care about the source of its seed?
< wumpus>
but I agree it's useful to have fast deterministic random for some things, I manually rolled a LFSR in one place in the tests to quickly generate deterministic "random" numbers
< jeremyrubin>
cfields: fine too -- just having something convenient to use for this purpose is all I care for!
< jeremyrubin>
The cuckoocache tests test hit rate against deterministic data, so if the entropy changes then it *could* (unlikely) fail.
< jeremyrubin>
but it depends on the variance of the hit rate... so it would be annoying if it fails say 1/100 times due to unlucky entropy
< wumpus>
yes, that should ideally never happen
< wumpus>
tests that randomly fail are extremely discouraging
< jeremyrubin>
Indeed
< jeremyrubin>
travis-gate was really a sad week for bitcoin ;)
< cfields>
heh
< jeremyrubin>
Speaking of tests -- any chance for putting #9495 (bugfix) and #9497 (tests) in 0.14? The CheckQueue is a pretty big piece of consensus code with almost no test coverage on it... Then again, there's no reason these tests can't wait for 0.15 either :)
< jnewbery>
jonasschnelli: I've got several PRs that are very disruptive to the qa directory: #9577 #9657 #9768 and the completion of #9707 (which I haven't yet PR'ed). Can you hold off your reorganzation of the qa directory until those are merged? Perhaps open an issue now for discussion but hold off on PR'ing.