< wumpus>
9634 seems ready, it would be nice if it had a test though - this seems like something that could regress early if it's not caught by the tests
< wumpus>
it doesn't necessarily need to hold up the pull request, but there's not been a reply from the author yet
< BlueMatt>
wumpus: I'll take a look at doing that in a bit
< BlueMatt>
question, would anyone object to me trying to add something like https://github.com/TheBlueMatt/RelayNode/blob/next/c%2B%2B/preinclude.h to bitcoin? it makes atomics replaceable with either std::atomic or a custom thing which has valgrind annotations so that helgrind/drd actually work and dont spew spurious crap all the time
< BlueMatt>
means atomics get declared with DECLARE_ATOMIC instead of std::atomic x, though
< BlueMatt>
well, suppose they could be OUR_ATOMIC_OR_NOT, but either way
< sipa>
why use macros?
< BlueMatt>
because otherwise you override std::atomic?
< BlueMatt>
i mean you can say all atomics are now of type MACRO_ATOMIC_TYPE
< sipa>
no, if you're going to switch all places where atomocs are used to macros, why not switch them with a self defined class directly
< sipa>
and when you don't need the self defined type, have it be typedefed to be std::atomic
< BlueMatt>
yea, well same thing either way
< BlueMatt>
also, due to the presence of _GLIBCXX_SYNCHRONIZATION_HAPPENS_BEFORE, that file would need to be included everywhere prior to the inclusion of any std includes
< BlueMatt>
well, the atomics could be in a separate file
< BlueMatt>
but...
< isle2983>
please don't talk about coding style at the meeting
< isle2983>
I have some automation pieces coming togeher, but it might take a little while to get it done
< wumpus>
BlueMatt: wouldn't it be possible to do this with std::atomic? I like the move towards using standard primitives instead of self-defined ttpes
< isle2983>
I am working near-full-time on this in the short term
< isle2983>
a viable end objective might be a 'Nit Bot' that can analize the commits from a PR and detect a few categories of things
< isle2983>
it looks like you can load a github acct auth token into the TravisCI account as an env var
< isle2983>
so, securely the travisCI script can post content back to github
< wumpus>
are you sure that's worth spending so much work on? there are some much more pressing issues than code style
< wumpus>
we've considered using that but the TravisCI token stuff is easy to circumvent if it's used for testing pulls - the test script for the pull could just output the token or upload it somewhere
< isle2983>
I am not sure it is the ultimate thing, but there is a cost to the project - you spent 20 minutes talking about it last week
< isle2983>
and there is some annoyance at trivial pulls
< wumpus>
that's a rarity, usually it doesn't come up at all
< isle2983>
if it can be automated, it should be automated at some point
< isle2983>
I am also of the school that says code is written once, and read 100s of times afterwards, so make reading easy
< isle2983>
probably 100,000s of times afterwards for bitcoin
< wumpus>
BlueMatt: in general the more 'you need to use this special type' rules a project has, the harder it is to contribute to. Though making it easier to valgrind/helgrind is certainly a valuable goal.
< wumpus>
isle2983: all too true, but I don't think most of the (superficial) code style makes much difference to that. Having a sensible design and the logic and the reason for things documented/commented is what is important to understanding difficult code
< isle2983>
totally agreed, but I see that as all the more reason to filter out distractions so that attention can focus 100% on the important things
< wumpus>
focusing too much on moving around spaces and such takes away the focus from what really matters, and it's much easier so it easily crowds out deeper thinking
< wumpus>
well as I see it a 'style nit bot' would add distractions, not remove them
< gmaxwell>
talking about pointless stuff is good for community. :)
< wumpus>
unless it annoys people
< isle2983>
in the short term, perhaps. but in the long term it would train submitters to get the PR right offline. Using common tools and scripts also lets the coders just integrate it into their text editor so they don't have to think.
< wumpus>
there's two ways to go at this sanely: one is to use something like clang-format from the begining. For example in golang projects everyone uses the same formatter with the same style, preventing any disagreements about style. The other is to do best effort and just not worry too much about it. It's too late for the former.
< wumpus>
our goal is not to 'train submitters'
< wumpus>
it's just importing concerns that shouldn't be part of this, creating extra bureaucratic barriers
< BlueMatt>
wumpus: yea, I tend to agree, I'm not sure I really want to do it, but if valgrind doesnt get an update to actually support std::atomic I might have to do it
< gmaxwell>
wumpus: I wouldn't worry too much; thats what review is for.. and that is the kind of thing everyone will spot, no one should mind fixing up, especially since it would be visible in practice all over... Obviously preferable to not.
< wumpus>
gmaxwell: I do worry about it. We've had this problem in the past with a certain person commenting on every pull about e.g. sorting include headers, whitespace, and so on, and it was incredibly annoying
< gmaxwell>
ah, I was commenting more on the macro for atomics.
< gmaxwell>
Yes. It certantly doesn't work if everyeone doesn't support it, especially if its merely cosmetic.
< wumpus>
right, I'm not worried about using a custom type for std::atomic, it shouldn't be used that much anyway. THough it seems to be something a tool should just handle.
< gmaxwell>
but "make valgrind usable to help us find data-races" is a substantial boon I hope we could all support. :)
< wumpus>
sure
< wumpus>
though it seems reasonably important to have it work for std::atomic so that the tool can analyse all projects using it, without needing special support at that side
< cfields>
wumpus: i wonder if that one, specifically, gets solved with newer versions of std libraries. For ex, iirc newer libc++ adds attributes to std::mutex/std::lock, etc, so that we won't need the wrappers in threadsafety.h
< isle2983>
I am happy to talk more about style automation on side channels - or also hear suggestions for what else I could be working on. I am in the learning phase and am open to anything that needs an extra brain and can help the project. Yielding now for important topics. Thanks.
< wumpus>
isle2983: to be clear I'm not trying to bash your work, just trying to set expectation how these kind of things are received, so you don't spend a lot of work automating something that won't be used
< wumpus>
cfields: indeed it could also be improved from the side of the library
< Chris_Stewart_5>
I've found it to be a useful way to get familiar with the core codebase as well
< Chris_Stewart_5>
wrt to what wumpus said about improving/adding tests
< isle2983>
wumpus: hey no worries. I wouldn't want to work on a project that didn't have strong skepticism and pushback
< isle2983>
Chris_Stewart_5: cool, I will take a peek. Thanks.
< sipa>
BlueMatt: are relaxed reads from an atomic even recognizable from the binary?
< BlueMatt>
relaxed I dont know, but certainly most reads/writes show up in the st as load()/store()
< BlueMatt>
its unclear to me whether helgrind is just being overly conservative and reporting them anyway because they could still be logic-races, or whether helgrind also isnt taking into account the ordering there
< BlueMatt>
its also unclear to me whether libc++ is doing the right thing with _GLIBCXX_SYNCHRONIZATION_HAPPENS_BEFORE set in their atomics and I just dont have that mapped to helgrind's equivalent
< BlueMatt>
which may be an easy fix
< BlueMatt>
(but that hadnt previously fixed it months ago when I was doing that for the relay network code)
< bitcoin-git>
[bitcoin] TheBlueMatt opened pull request #9671: Fix super-unlikely race introduced in 236618061a445d2cb11e72 (master...2017-02-fix-initnode-race) https://github.com/bitcoin/bitcoin/pull/9671
< BlueMatt>
cfields: I blame you for ^, btw, you asked me to do it :p
< cfields>
BlueMatt: In my defense I'm pretty sure i only said that it was ugly and that it should be cleaned up later, but I didn't complain when you went ahead and moved it either. So I'll take the blame :)
< BlueMatt>
lol, yea, well /someone/ should have bothered to look at the implications :(
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu Feb 2 19:00:02 2017 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< gmaxwell>
fixing the fact that it's very easy to fail to rescan anything, when you thought it was... does.
< wumpus>
yes undocumented or could add a "warning: experimental, API will likely change next release" in any case too
< jonasschnelli>
Or we just fix 9491... seems not very complex?
< jonasschnelli>
Can fix in rc2 if it's to late for monday?
< wumpus>
sure
< wumpus>
but there's no guarantee there is a rc2
< gmaxwell>
okay, lets see where that goes in the next couple days.
< wumpus>
I don't know how hard it is? it seems to have caused quite a discussion but no fix
< luke-jr>
importmulti seems akin to importprivkey which shouldn't be used by users anyway?
< gmaxwell>
We can hide it right before cutting RC1 if nothing else.
< wumpus>
yes
< gmaxwell>
ashame, as it's a nice improvement.
< sipa>
i think the fix would be easy?
< gmaxwell>
sure, that is why I said lets see.
< BlueMatt>
who is working on it?
< achow101>
can't you just change the default timestamp to be 0?
< gmaxwell>
but we have a fallback if it doesn't get fixed.
< BlueMatt>
"lets see" only works if someone does it :p
< gmaxwell>
achow101: then there is no easy way to express now.
< jonasschnelli>
Would a fix where we set the importmulti timestamp to 0 instead of "now" do it for 0.14?
< luke-jr>
gmaxwell: using time() from your OS?
< jonasschnelli>
*default timestamp
< achow101>
-1 for "now"
< wumpus>
or have no default at all and require a time to be specified
< jonasschnelli>
wumpus: +1
< luke-jr>
wumpus: no default at all is nice since it allows a default to be chosen later
< jonasschnelli>
0 as timestamp is very inefficient.
< gmaxwell>
Lets not hash it out here, there is an issue.
< jonasschnelli>
Okay. Lets comment there. Agree with gmaxwell
< gmaxwell>
I agree with jonasschnelli in the sense there that we really have to stop assuming a full rescan is possible.
< wumpus>
good point, yes
< jonasschnelli>
Is also very inefficient if you have pruned or run hybrid SPV
< wumpus>
it certainly shouldn't be the default
< gmaxwell>
It takes many hours on my normal development system, and is still quite slow even on the fastest hardware available. But avoiding the rescan takes second seat to surprising the user. :)
< wumpus>
it's inefficient and lazy
< gmaxwell>
in my view, except for certan recover operations that are infrequently done-- rescan effectively doesn't work anymore (takes more time than converting your entire usage to a third party api...)
< wumpus>
users of the API should be encouraged to keep track of key birthdates
< gmaxwell>
if we define a new private key format in the not so far future, we should make sure its string clearly integrates a birthdate. :P
< BlueMatt>
ok, so discuss on the issue....next topic?
< wumpus>
a full rescan is indeed only something that should be done for infrequent recovery reasons
< wumpus>
no other topics?
< MarcoFalke>
shortest meeting ever
< wumpus>
I had expected heated debates on what to include last-minute in 0.14 and why to delay the rc, what a disappointment! </s>
< BlueMatt>
great, lets get 0.14 done so I can get back to writing code :)
< jonasschnelli>
Heh.
< sdaftuar>
let's talk about code style again
< BlueMatt>
wumpus: I vote we push it back a month so we can do all the things we wanted to a month ago :p
< wumpus>
BlueMatt: lol!
< BlueMatt>
wait, i had something to talk about re: cde style
< BlueMatt>
hum
< gmaxwell>
BlueMatt: die
< sdaftuar>
i'll get the baseball bat
< BlueMatt>
oh, auto
< jonasschnelli>
Bumpfee: is there a reason why the logic is in the rpcwallet.cpp and not in wallet.cpp?
< jonasschnelli>
Makes it really hard to use in the gui...
< BlueMatt>
jonasschnelli: please move it, agreed
< luke-jr>
jonasschnelli: it can be moved
< sdaftuar>
jonasschnelli: i think we can refactor as needed
< luke-jr>
in 0.15*
< wumpus>
jonasschnelli: because it's only used in rpcwallet.cpp, if you need it in a more general place move it
< jonasschnelli>
Okay.
< sipa>
BlueMatt: i am strongly in favor of auto.
< * sipa>
hides
< wumpus>
jonasschnelli: although moving everything to wallet.cpp isn't very nice either, we should refactor the wallet code some day
< luke-jr>
I did suggest it earlier, but didn't seem like a blocker for merging
< wumpus>
I'm also in favor of auto
< BlueMatt>
sipa: it makes certain review much, much harder (I often grep for "everywhere X is used")
< luke-jr>
wumpus: well, it's wallet code..
< BlueMatt>
and have already missed things as a result
< wumpus>
luke-jr: so? not all wallet code needs to be in one file
< wumpus>
forbidding auto is just masochism
< jonasschnelli>
wumpus: yes. Thats a good point.
< BlueMatt>
wumpus: i wasnt voding forbidding it
< sipa>
BlueMatt: introduce an incompatible change to the type, and recompile. tadaa, all places it is used
< BlueMatt>
only carefully considering its use
< cfields>
sipa: same. BlueMatt: maybe paste the thread in question?
< luke-jr>
I like auto when the type is implied by some other type; eg, instead of xyz::value_type
< jtimon>
yeah, but not forbidding it doesn't mean recommending it always either
< wumpus>
we have a whole src/wallet directory which could have tons of different implementation files for different facets of the wallet, instead of stashing it all into one file
< BlueMatt>
(I believe gmaxwell's comment there was intedned for a different line)
< jonasschnelli>
yes. Stuff like coin selection should be more modular
< wumpus>
sure, as with any use of any c++ statement, use of auto should be measured
< MarcoFalke>
Lets do it after priority removal
< MarcoFalke>
Otherwise we step on each others toes
< wumpus>
if you have some specific cases where it's bad to use auto, please document them
< BlueMatt>
wumpus: mostly only things that are /actually/ a mile of text to type, imo
< sipa>
BlueMatt: and not needing to change things all over the place when you turn a tuple into a struct
< sipa>
or add a wrapper
< BlueMatt>
sipa: I have no problem reviewing sed-based changes
< BlueMatt>
in fact prefer that
< wumpus>
BlueMatt: there's plenty of those - c++ is overly verbose, auto is a great advancement
< sipa>
they're still annoying to fo
< BlueMatt>
since I'm gonna go read every single place the change effected anyway
< BlueMatt>
to review
< sipa>
*to do
< sipa>
and of course, let's consider on a case by case basis
< wumpus>
right
< BlueMatt>
wumpus: sure, iterators in iterators, np
< BlueMatt>
yea, ok, whatever, I'll shut up
< sipa>
but in my own preference, that is overwhelmingly the case
< cfields>
well the specific case here is for loops. "for (auto& foo : bar)"
< MarcoFalke>
Agree with BlueMatt, that auto should not be used unless necessary.
< cfields>
any reason not to use auto there?
< jtimon>
BlueMatt: the question is, do you have a general advice on when not to use auto?
< wumpus>
it's never *necessary* auto is just nice
< BlueMatt>
cfields: yes, so I can grep and review if the type's behavior changes in some way
< BlueMatt>
jtimon: personally, if the type really, really doesnt matter
< luke-jr>
cfields: if it's liable to produce bad results with bar changing under it
< BlueMatt>
(which means very rarely use it)
< jtimon>
BlueMatt: I'm afraid "doesn't matter" it's too vague here
< wumpus>
I don't think this is going anywhere, too much isagreement
< BlueMatt>
eg if you're taking an iterator and passing it through to another function
< wumpus>
any other topics?
< wumpus>
BlueMatt: function arguments can't use auto, right?
< sipa>
indeed
< sipa>
c++14 and later introduce some auto types in lambdas
< BlueMatt>
wumpus: correct, but eg doing auto it = map.find(thing); if (it != ma.end()) DoThingWith(*it);
< BlueMatt>
is like not a problem
< BlueMatt>
auto it = map.find(thing); if (it != ma.end()) ILikePonies(it->second.rainbows); I do not like
< sipa>
BlueMatt: how is that different from a for (const auto& x : container) {}
< BlueMatt>
sipa: because in the specific case here the thing in the loop is not defined to take a specific type
< BlueMatt>
it is templated
< jtimon>
my question was, do you have a deductive method for finding the not ok cases instead of an inductive one for the "not a problem cases"?
< sipa>
you can see that as an oblivious loop with iterators, and passing *it to a function that is tje body of the loop
< BlueMatt>
jtimon: <BlueMatt> jtimon: personally, if the type really, really doesnt matter
< sipa>
i see your point, but i don't think it weighs up against the benefitd
< sipa>
*benefits
< wumpus>
if you're iterating over some container, the type of container usually really doesn't matter, unless you make specific assumptions (but then you'd generally not be using a range for loop in the first place)
< BlueMatt>
wumpus: imo if you are ever actually dereferencing the type you should not use auto
< sipa>
BlueMatt: your own example dereferences...
< BlueMatt>
if you're dereferencing the iterator to eg a pair or just taking the element and passing it to something else, ok
< BlueMatt>
but if you're dereferencing it and accessing something inside it, no
< sipa>
then we might as well not use it at all, i think
< jtimon>
ok, I think I get what you mean by "doesn't matter" now
< BlueMatt>
sipa: there are many places where you might do for (auto& thing: list) ActOn(thing);
< BlueMatt>
thats reasonable
< sipa>
requiring programmers to spell out redundant information just so you can grep for it seems extreme to me
< gmaxwell>
sipa: so functions shouldn't have prototypes? :)
< BlueMatt>
yes, I didnt expect people to agree with me...I have extreme distaste for auto, personally
< wumpus>
yes, that' extreme, and not going tohhappen. Just use smarter tools.
< BlueMatt>
wumpus: suggestions?
< * BlueMatt>
would love a grep --allusesoftype thing
< wumpus>
it should be fairly easy to implement using clang's parser, would be surprised if it doesn't exist
< gmaxwell>
There is another side to it is that auto enables you to write code that acts on a type while having no idea of the type yourself. Which is safe 99% of the time and deadly the rest.
< gmaxwell>
because in C++ not all operations which are catgorically unsafe on a type are actually stopped by typechecking. :(
< gmaxwell>
I have an auto to a container... and then I extract an auto to an iterator on it and erase things. Is my code guilty of the sin of using an invalidated iterator? It depends on what container was in use, and that was hid by auto...
< gmaxwell>
But... that sort of thing is an edge case, I'd love to see a realistic list of where auto is likely to cause problems, just to keep it in mind.
< wumpus>
right - just keep it in mind while reviewing
< wumpus>
and if there are well-defined cases where auto is dangerous, they should be documented in the developer notes
< BlueMatt>
ehh, ok, well I go read all of wallet half the time reviewing wallet changes, i guess I'll just start doing that for net, too :p
< BlueMatt>
(not a bad thing, that)
< gmaxwell>
unfortunately, auto is most interesting when you have some horrible complex signature. But those are the cases where it is also more of an issue.
< cfields>
gmaxwell: for(auto& : foo) doesn't give you an iterator though, just a reference. So imo that should be highly preferred when possible to avoid your example.
< wumpus>
well no, it's most interesting for bog-standard loops, 99% of the cases. If you're doing anything horribly complex, that's probably where you should be careful
< jtimon>
BlueMatt: we can agree that auto is totally fine for unittests too, right? :p
< cfields>
(preferred over auto foo = bar.begin(), that is)
< gmaxwell>
wumpus: well my point is that stating the type explicitly is just as easy as auto when it's simple and obvious.
< sipa>
gmaxwell: when you have some horrid complex type signature, best practice is to introduce a typedef for it... that also results in succint usage, and lacks the review concerns that BlueMatt has i think
< jtimon>
I agree it removes clarity some times
< wumpus>
gmaxwell: it's *easy* but the point is to avoid unnecessary verbosity/typing, not so you can forget the type
< BlueMatt>
sipa: yes, agreed, also means dont use auto in place, which some people like to do
< jtimon>
but I don't have a clear criterion on when to use it like matt
< BlueMatt>
wumpus: I'm generally 100% in favor of extra verbosity
< wumpus>
e.g. to avoid having to type std::vector<std::Strring> for the zillionth time
< gmaxwell>
fking java programmers. :P
< wumpus>
BlueMatt: go use java
< BlueMatt>
extra verbosity generally means less magic, which makes review easier
< BlueMatt>
lol, i expected that....
< gmaxwell>
(though on type signatures, I usually also prefer being explicit more often)
< wumpus>
BlueMatt: that's not categorically true, more verbosity also means more distraction
< sipa>
agree
< BlueMatt>
wumpus: well, ok, more verbosity as long as it actually provides information
< sipa>
nobody actually looks at what large type definitions contain
< wumpus>
having lot's of boilerplate does *not* equal easier review
< BlueMatt>
public static void main(String[] args) {} probably doesnt provide more information
< wumpus>
anyhow
< BlueMatt>
sipa: I do!
< wumpus>
any other topics? this is going the wrong way
< sipa>
haha
< luke-jr>
lol
< BlueMatt>
soooo...endmeeting?
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Feb 2 19:40:26 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< jtimon>
also for "we can fix that after the branch" types of things
< MarcoFalke>
jtimon: master is for testing, release branches are for production
< jtimon>
MarcoFalke: I know
< gmaxwell>
The asserts don't leve enough record in any case, during development.
< MarcoFalke>
jtimon: The thing is you may want asserts during testing, but you may not want them in production
< gmaxwell>
I've had asserts fire and have had no idea which assert fired or why, only that my daemon went away. So a crash was introduced but we didn't even learn from it.
< MarcoFalke>
jtimon: We can not compile without asserts, so this is the only way?
< jtimon>
MarcoFalke: perhaps something like if(debug) ?
< gmaxwell>
(moreover, differences in behavior between testing and production exposes you to bugs that the testing code fixes)
< gmaxwell>
e.g. when an asserts check has a side effect.
< gmaxwell>
In any case, asserts around the net stuff are also special because the testing there is woefully inadequate right now.
< gmaxwell>
As evidenced by the fact that the version assert was there for months and only triggered a couple times, but yet was still triggerable!
< jtimon>
yeah, I'm talking more generally about the "we can fix that after the branch" or let's do A and master and B in 0.14 attitude
< luke-jr>
#9619 has plenty of ACKs; shall I go add test cases, or merge this and PR test cases separate?
< gribble>
https://github.com/bitcoin/bitcoin/issues/9619 | Bugfix: RPC/Mining: GBT should return 1 MB sizelimit before segwit activates by luke-jr · Pull Request #9619 · bitcoin/bitcoin · GitHub
< BlueMatt>
I do believe we need to start talking about making our assert infrastructure better - building test bins by default with DEBUG_LOCKORDER and many more asserts, with most of those asserts just printing warnings in production
< jtimon>
yeah, and that doesn't require to have different code, maybe just change the value of a constant or something
< jtimon>
or do all that stuff when -debug
< bitcoin-git>
[bitcoin] isle2983 closed pull request #9459: Improvements to copyright_header.py and some minor copyright header tweaks. (master...PR-copyright-script-improve) https://github.com/bitcoin/bitcoin/pull/9459
< bitcoin-git>
[bitcoin] isle2983 closed pull request #9603: Add basic_style.py to automate some style checking. (master...PR-basic-style) https://github.com/bitcoin/bitcoin/pull/9603
< sipa>
jtimon: indeed. my preference is that anytime you're touching code (except simple move-onlys), you adapt the style of the code
< sipa>
but no big refactors changing the style all over
< jtimon>
right, my point is I would prefer that our rules for style was something that we're actually closer to achieve, even if that's something less strict
< jtimon>
this is an invitation for someone to create a PR correcting some of those cases
< isle2983>
it seems strange to me that the style is a moving target. is there some nuance that I am missing?
< isle2983>
I would think that one of the off-the-shelf choices would be best.
< sipa>
isle2983: the original satoshi code used a very arcane style, which many people dislike, and has in practice not been followed by contributors after satoshi left
< sipa>
at some point we formalized the effective style people were using in the developer notes, but reviewers didn't actually enforce it
< sipa>
plus, there was a strong "mimick the style around what you're editing" tendency, which doesn't really help converging
< sipa>
i hope that going forward we start enforcing it
< isle2983>
I am happy with that as a starting place
< jtimon>
IMO, style rules aren't so useful until you start to enforce it with the help of automatic tools in the whole project, that's the only way you can really stop talking about style, which is the goal of having style rules in the first place, right?
< gmaxwell>
sipa: clang-format (latest versions) can enforce that rule.
< gmaxwell>
jtimon: I don't agree. Code style existed long before tools to do anything special with it.
< jtimon>
gmaxwell: thanks for confirming, that's what I thought, including the exception for the single line ifs
< gmaxwell>
I think automatic tools often handicap style discussions, because they enforce it stupidly to the point where they can irritate people and cause general opposition to standards.
< gmaxwell>
jtimon: yes, including the exception, according to the docs. I have not tested it.
< jtimon>
well, code style without automatic rules may save you some style discussions, but not all
< gmaxwell>
(I don't have a current version of clang on my laptop -- another challenge with formating tools, they change.)
< jtimon>
yep
< gmaxwell>
jtimon: they give people a way to answer the question without first asking everyone else all the time. :)
< jtimon>
absolutely
< gmaxwell>
(I'm not opposed to tools, but they work MUcH better for single developers and single companies with rigidly enforced development workstation configs.. than they do for big projects)
< jtimon>
true that, my very positive experience where with enforced development confgs per project, the project that had discussions about syle were those not enforcing the style automatically
< jtimon>
s/experience where/experiences were
< isle2983>
the trend does seem to be towards containerized environments for builds there the dependencies can be managed. That might be wrong for bitcoin, however.
< luke-jr>
it wouldn't be the first time we have an untestable tree merged in history
< luke-jr>
it's a potential security issue to keep in mind though
< luke-jr>
(bisecting unsigned commits)
< MarcoFalke>
Well, I hope when people review, they check that the intermediate commits don't add random binary blobs
< gmaxwell>
MarcoFalke: code that totally backdoors your machine could be a couple lines of shellscript. And the concern there is just that you pull and some is slipped in and you run in bisect... it's worried me before, but didn't seem like there was anything to do over it.
< MarcoFalke>
Still should be part of review to ensure this does not happen
< gmaxwell>
it's not something review can control.
< kanzure>
hi am i late
< gmaxwell>
I do wish you could check in a .gitbisectuntestable which has a list of commits bisect should just skip over.
< MarcoFalke>
Please explain why review can not control it
< sipa>
gmaxwell, MarcoFalke: write a script that produces a list of all unsigned commits, and feed those to 'git bisect skip
< sipa>
?
< jtimon>
BlueMatt: at this point, should I just close #9634 as included in #9650 ?
< gmaxwell>
MarcoFalke: Because compromised code can hide things from reviewers. And there is no guarentee that things that hit the tree have been reviewed (particularly if a commiter is compromised). So if you get compromised you might merge bad commits. You can't see they're bad because the compromise hides them. Other people don't see they're bad because they only review the ultimate commit if at all.
< gmaxwell>
.. and becuase once they run something in a bad commit they become compromised and can't see it either.
< jtimon>
BlueMatt: ok, thanks for the tests!
< luke-jr>
all the more reason to sandbox dev
< gmaxwell>
especially since we don't require that PR that we merge be based on the current tip (it would be unreasnable to do so), the intermediate commits will often not match exactly what anyone has reviewed.
< bitcoin-git>
[bitcoin] jtimon closed pull request #9634: Fail in DecodeHexTx if there is extra data at the end (master...upstream-fail-decode-tx) https://github.com/bitcoin/bitcoin/pull/9634
< gmaxwell>
Even where the final result does.
< BlueMatt>
jtimon: thanks for catching that I forgot to upstream this!
< jtimon>
np
< luke-jr>
it'd probably be more practical to review if we used merge commits instead of rebasing
< MarcoFalke>
Which is why I proposed we don't corrupt merge commits
< MarcoFalke>
> Other people don't see they're bad because they only review the ultimate commit if at all.
< MarcoFalke>
Review should always be done commit-by-commit
< gmaxwell>
MarcoFalke: conflicts though is not the same as no difference at all.
< gmaxwell>
Bascially if you require there be no difference then we must require every PR rebase and re-review every time another PR is merged that touches the same files (or at least functions).
< gmaxwell>
Otherwise you can arrange the automatic resolution to add vulnerabilities.
< gmaxwell>
and already people will review commits, but not review that what is ultimately merged were the same commits exactly (usually aren't, due to other changes)
< MarcoFalke>
I don't understand how you end up with two different results when you merge the exact same commits.
< MarcoFalke>
I mean you get a different commit hash when you don't adjust the time to time-of-merge and author to merge-commit-author etc
< MarcoFalke>
but it should be possible to replay all merge commits and compare them to what peoples eyes reviewed
< jtimon>
BlueMatt: you missed one garbate in 9650 :p
< BlueMatt>
oh ffs, leave my spelling alone!
< BlueMatt>
/s
< gmaxwell>
MarcoFalke: If I review PR X which is on head Y (or which I merged to Q) that is not the same as reviewing PR X merged against Z. (particularly in adversarial condictions)
< gmaxwell>
I think this is getting too abstract, for this discussion I think it suffices to point out that people will pull code into their local branches before reviewing it, because pulling it into their branch is how they go about reviewing it.
< MarcoFalke>
Reviewers don't commit to "PR X merged against Z", they commit to "PR X".
< MarcoFalke>
They can only commit to the merge commit when the merge actually happened in the master branch
< MarcoFalke>
I mean I can merge a commit_A locally for testing, but when I publish my review I don't refer to my local merge commit by to commit_A
< gmaxwell>
You're saying the same thing as me--- I think. Reviewers review PR X and so cannot be counted on strongly to catch malicious behavior in X which is only exposed by its merge in Z.
< MarcoFalke>
/by/but/
< gmaxwell>
MarcoFalke: right, and so your review /may/ be ineffective, particularly if X was maliciously constructed.
< MarcoFalke>
Oh I think we were talking about two completely unrelated things, I think. You were raising concerns about possible exploits that only arise after gits merge algorithm processed them. I was talking about maintainers appending commit/ ammending commits/ editing merge commits or reviewers skipping individual commits of pull request which happen to contain a +++ exploit.sh and ---exploit.sh
< gmaxwell>
Right. and I am saying that I do not believe people review intermeadate commits in merges, because doing so cannot show issues that arose through the merge. They look at PRs and at end results. If you slip in extra commits that do not change the end results, I believe it is unlikely to get noticed.
< gmaxwell>
Perhaps I'm wrong.
< MarcoFalke>
I don't think you are wrong, so we should work towards making it easy such that a uncompromised machine with a helper script and some verified keys can validate the current state of the bitcoin-git repo
< luke-jr>
ideally, but I think we need to prioritise getting reviews done over making reviews more effort
< MarcoFalke>
Yeah, that is the downside.
< bitcoin-git>
[bitcoin] ryanofsky opened pull request #9673: Set correct metadata on bumpfee wallet transactions (master...pr/bumpfee-meta) https://github.com/bitcoin/bitcoin/pull/9673