< gmaxwell>
man, should have locked that right away.
< gmaxwell>
Someday I will invent the devices that allows one to stab someone in the face over the internet, but today is not that day.
< sipa>
killall -SIGECUTE
< kallewoof>
luke-jr: Regarding comment on #13756 -- you say "This doesn't actually fully solve #10065, since transactions received with a dirty address are still shown in the GUI / RPC". Are you referring to the case where someone sends funds to destination X, and then later send funds again to destination X before I spend from X? Why is that a problem? X is not dirty unless I spend from it once, in which case the second send WILL be
< kallewoof>
Oh wait, you're talking specifically about the GUI/RPC. Yes, I guess this only applies to the coin select algo...
< sipa>
kallewoof: i guess luke-jr wants it to not even treat a second receive at the same address as a valid incoming payment
< kallewoof>
It feels to me like this needs to be split up into multiple PR's to be manageable in size. I wonder what the MVP is for an acceptable merge for people -- a lot of talk is focused on GUI stuff, which I felt would come later.
< kallewoof>
sipa: Yeah. I mean, it won't be, if you use the built in send mechanism since both CLI and QT use the coin selection methods which will exclude these.
< sipa>
kallewoof: i assume he means for listtransactions/getbalance/...
< kallewoof>
Right.
< sipa>
(i have very little opinion about the matter)
< kallewoof>
Maybe I should just make multiple PR's that stack on top of each other. I do intend to address the RPC side of things at least, and perhaps take a stab at QT, but I tend to fail whenever I touch that code.
< luke-jr>
kallewoof: yes, multiple steps seems like a good approach
< luke-jr>
kallewoof: and "reused" *does* convey it's to be avoided, at least as an internal term (which IMO is all this should be)
< kallewoof>
You mean we should use a different term for the actual user interface? Doesn't that make things unnecessarily confusing?
< luke-jr>
kallewoof: the UI should just show reused transactions as eternally unconfirmed until spent ;)
< luke-jr>
notated somewhere that they won't confirm normally
< luke-jr>
(and perhaps coin selection should actively try to spend them..)
< kallewoof>
Wait, coin selection should try to spend them? I thought the whole idea was that coin selection should never spend them.
< kallewoof>
And eternally unconfirmed sounds like it will result in ∞ questions from people
< gmaxwell>
luke-jr: I think showing them as unconfirmed is going to far, especially instantly.
< gmaxwell>
better to first show some warnings on them, this will cause people to complain and change behavior.
< gmaxwell>
kallewoof: there are some cases where they would be okay to spend, e.g. if we can spend them by themselves with no other inputs (esp with no change).
< gmaxwell>
This is with the idea that the main thing we're trying to do is prevent the snowball effect where most of your wallet is linked togeather.
< kallewoof>
Yeah, I was thinking about that before. It seems like you care about privacy differently in different situations. E.g. donations to a charity from a dirty address sounds like a great idea.
< kallewoof>
Generally, it feels like a 'categorization mechanism' (accounts done right) would be quite useful. Spend A coins together as appropriate. Mix A and B only if funds are lacking, and put resulting change in category B, not A. Etc etc.
< luke-jr>
kallewoof: if we never spend them, then they shouldn't be shown at all, or should be shown as conflicted or similar
< luke-jr>
kallewoof: by spending them, we remove the security risk
< luke-jr>
(which is much less than the privacy risk, so perhaps not the prevailing motivation on ideal behaviour here)
< kallewoof>
I assume this is almost always slightly-above-dust amounts, so the most common scenario will be to simply not touch it, no?
< kallewoof>
It feels odd to proactively try to spend it.
< luke-jr>
dust spam could just be ignored; the cases I'm thinking of are when the sender sends twice
< kallewoof>
I'm sort of optimizing for the case where someone is actively trying to track you by purposefully spamming you with tiny outputs so they can see where your coins are going. That seems quite different from what you're talking about..
< kallewoof>
And sounds like the proper reaction is different too. Should the case 'reused, tiny' be treated differently from the case 'reused, non-tiny'? How do you define 'non-tiny'? (Dust threshold * X I guess)..
< kallewoof>
I am currently adding 'allow_dirty' bools to various methods. Maybe it would make more sense to have a three-option state instead (mixed, clean, dirty), where mixed=ignore dirty state, clean=only dirty=false, dirty=only dirty=true.
< Fuzzbawls>
Question: (please ping me directly with answer) Given the recent acquisition of GitHub by Microsoft, is a relocation of the bitcoin source code to an alternative platform (like GitLab) being considered at all? If so, has anyone been actively working towards porting the `.travis.yml` file over to an alternative CI provider? I've had mixed results with such a port; some features can be duplicated/represented, and some cannot (like the $TRAVIS_COMMIT
< Fuzzbawls>
_RANGE and the $TRAVIS_EVENT_TYPE environment variables, for example)
< kallewoof>
I have opened #13801 as an alternative, which replaces the dirty flag with a dest_filter=mixed/clean/dirty. More fine grained control, but slightly bigger diff. Will close one or the other based on feedback.
< wumpus>
I really don't like all these PRs that just brush the code a bit
< wumpus>
all the diff noise makes it harder to keep track of what is changing, and if it's not a clear win in, say, avoiding a class of bugs, I don't think it's generally worth it
< wumpus>
it also creates endless motivations to create new PRs
< wumpus>
please stop it
< fanquake>
wumpus I think I agree.
< fanquake>
There are quite a PRs with feedback/questions being left outstanding while new PRs are continually being opened (by the same author etc).
< wumpus>
fanquake: thanks; I mean it's obviously not black and white, some changes make more sense than others, but sometimes it looks like they're just being dished out for the sake of making changes
< wumpus>
I mean I'm sure there's arguments for it, but shouldn't that first be discussed, then maybe added tothe code style for *new code*
< wumpus>
not just roll over the entire code, out of the blue, and change something that we weren't even aware about before
< fanquake>
Yea. This one was sort of similar #13795.
< gribble>
https://github.com/bitcoin/bitcoin/issues/13795 | build: Add missing [[noreturn]] to handleRunawayException(...). Enable -Wsuggest-attribute=noreturn if available. by practicalswift · Pull Request #13795 · bitcoin/bitcoin · GitHub
< wumpus>
yes
< fanquake>
No feature/build flag I hadn't really seen before. One build failures, and it gets closed with no more discussion?
< fanquake>
*A feature..
< fanquake>
Surely at least some followup or an explanation of what didn't work would be handy, even just for a future PR which might try doing the same thing.
< wumpus>
the problem is that everyone is overworked, let's avoid creating unnecessary work out of the blue
< wumpus>
that's basically my entire criticism
< fanquake>
Yep, that's a fair call.
< fanquake>
It's kinda felt like the repo has been "slowing down", over the past month or two
< fanquake>
If not a longer period than that
< wumpus>
I'm not sure that is true
< fanquake>
Maybe it's just the fact that there is a lot of different work going on that makes it harder to gauge progress
< fanquake>
The point of applying to new code is good. We've seen continual cases of follow up changes/fixing things up "just after" they were merged.
< wumpus>
right!
< kallewoof>
I agree gaining consensus for code style update (via PR and (concept) ACK's w/ merge) should be required before making changes unless the change is an obvious improvement that everyone needs. However, I also believe that we should automate EVERYTHING that can be automated, even if it is only slightly beneficial.
< kallewoof>
Automating PR review seems like an obvious win.
< wumpus>
at least I'm not arguing against automated PR review
< wumpus>
I think checking for things that actually makes sense is good
< luke-jr>
kallewoof: PR review cannot be automated..
< wumpus>
but not adding vague concerns all the time that haven't actually resulted in bugs nor are likely to result in such, we're trying to build functional code here, the concern is not perfect style according to some person's preferences
< fanquake>
kallewoof I agree re automation. At least in the case of the Draht bot, as it has started improving/gotten a bit less noisy, I've begun to enjoy the merge conflict notifications, as well as nearly automated? gitian builds.
< fanquake>
However posting a comment to say that 15 PRs all conflict with each other is probably getting towards the less valuable end, heh.
< wumpus>
luke-jr: I think he means the linters, so the basic trivial stuff
< wumpus>
if only real code review could be automated!
< kallewoof>
wumpus: I admit I haven't seen the PR in detail, I was mostly talking about past experience :)
< kallewoof>
and yes, I am talking about the review that you can automate, not all review.
< wumpus>
kallewoof: I think MarcoFalke's approach is better than practicalswifts in this regard; DrahtBot has a PR that updates various known things before the 0.17 branch, practicalswift on the other hand creates PR after PR after PR changing things all over the code base
< wumpus>
I think agreeing to make some such changes (certainly if they're alrady part of the coding guidelines) before a release is okay, at least it's a rare thing
< fanquake>
Something like #13802 might be just that sort of thing, but I've added there that it should really be added the the build system if possible. Otherwise could lead to endless followup.
< fanquake>
i.e sweeping change just before a release
< wumpus>
my concern is with sweeping changes that come out of the blue
< wumpus>
suddenly there's this completely new thing you should care about and we should update the entire cod efor!
< wumpus>
please, let's work on issues that affect users
< kallewoof>
Ideally, if a PR will plug a potential issue later down the road, it should be encouraged, but it can be hard to judge. I think a way to solve this is to put the weight on the PR creator -- find compelling and easy to follow reasons for why the PR is necessary, or we all agree to concept NACK saying we don't see the reason. Maybe even add that to the contributor guidelines.
< wumpus>
yes, if there is a good rationale, we have a way to introduce that (through the coding guidelines), I agree
< wumpus>
that's exactly what I said in my post, too
< kallewoof>
*nod*
< kallewoof>
wumpus: Does #13803 convey the right message?
< wumpus>
kallewoof: I think so! wouldn't a better place be CONTRIBUTING.md though?
< kallewoof>
I was unsure about that. I have no strong feelings about it, personally, so can move it.
< kallewoof>
Actually, it fits better in contributing, I think. Moving it.
< wumpus>
that's where people automatically get linked afaik
< kallewoof>
Moved
< fanquake>
ken2812221 Are you following any of the upstream Windows/leveldb work? I've seen a few different PRs that also seem to be working on the unicode problems that you are.
< fanquake>
However progress on them seems to be stagnating.
< ken2812221>
fanquake: No, I haven't seen those yet.
< ken2812221>
google might not want to port leveldb to Windows.
< ken2812221>
I just do the minimum changes base on current env_win.cc
< fanquake>
np. Just mentioned as it was worth a look incase you could cherry-pick anything useful.
< ken2812221>
Thanks, will take a look.
< fanquake>
wumpus You can probably merge 13803 straight in. 13554 also ready.
< wumpus>
thanks!
< fanquake>
Also 13797, backporting to 0.16
< kallewoof>
wumpus: mind if I fix commit message s/developer/contributor/ on 13803? don't wanna touch if you intend to merge it now.
< kallewoof>
Actually promag nit is valid so gonna fix that too. Apologies.
< wumpus>
kallewoof: for sure!
< wumpus>
kallewoof: this should be open for a while for discussion probably nayway
< kallewoof>
Yes, good point.
< kallewoof>
Although practicalswift did already ACK it. Not sure if anyone else is affected (apart from myself, of course.. I mean, this obviously means #12879 is not gonna see the light of day, ever).
< satwo>
Is there a more detailed guide (than what is currently found in CONTRIBUTING.md) that explains, step-by-step, the proper testing of a PR? I would assume at minimum: compile and run the patch set, make sure the intended behavior is present, make sure no unintended behaviors have been introduced (is there a systematic way to approach this?), and run (all, or just relevant?) unit tests.
< fanquake>
"compile and run the patch set" Quite often this needs to be done across multiple OS's, as occasionally a something will break a *BSD build, or Windows etc
< fanquake>
Running all the tests suites (--extended), running any linters etc.
< fanquake>
However what your testing/checking is always dependant on the actual change.
< fanquake>
There was at one point a bitcoin test cases/writeups repo, but I don't think it ever got much traction
< wumpus>
it depends on the change; if it is aspecific, say a refactoring, then running the test framework (both unit and functional) should be enough
< wumpus>
if it is specific, for example you're adding an API call, then you need to add a functional test for that call
< wumpus>
for more complex internal functionality you'd want to add unit tests
< satwo>
kallewoof: ha! That must have been quite the learning experience, if nothing else :)
< satwo>
fanquake: That was something I was wondering about: how does one know when a change should be tested on different platforms, and when testing on any single platform is sufficient?
< kallewoof>
Yeah I enjoyed it. :) LImited use but still
< gmaxwell>
ISTM that the project has sped up in the last year in terms of changes, but I don't think that the number of interesting new features, performance improvements, or bug fixes have increased at the same rate which sometimes just makes it seem like more work to keep up but less interesting.
< wumpus>
satwo: in any case if you are working on something, just ask, and people here can probably answer what you need
< gmaxwell>
satwo: In terms of "systematic way" -- the systematic part should be largely implemented by the automated tests.
< gmaxwell>
So for testing a PR: run the automated tests, test your own use cases, and test whatever you can come up with specific to the PR that isn't automated.
< satwo>
wumpus: thanks! I definitely need to familiarize myself with functional vs unit tests in bitcoin. I'll continue to come here to ask my noobish questions.
< jonasschnelli>
MarcoFalke: there are two merge commits in #13804
< kallewoof>
satwo: functional = start up one or several nodes and try stuff out by calling them via RPC. unit = actually call C++ methods directly and check if they behave right
< fanquake>
satwo: Here's one example of a PR that "silently" broke builds on FreeBSD 10 9598.
< fanquake>
There gui can also get somewhat limited testing, especially on Windows. Although, that stems more from so few of the developers, at least working on Core, using Windows.
< fanquake>
However that might seem to be changing if you take into account recent PRs.
< wumpus>
breaking FreeBSD is a good way to make me angry :)
< satwo>
gmaxwell: thanks for the clarification! Very helpful.
< wumpus>
but that doesn't happen that often, usually with build system changes, those I test on *BSD first before merging
< satwo>
kallewoof: Thanks for the clarification! Pretty straightforward yet I'd never considered the difference until today.
< satwo>
fanquake: thanks for the PR reference, very interesting. I do have Windows running on Parallels so I could certainly help out on that front.
< jamesob>
did we recently bump the required version of protobuf?
< wumpus>
jamesob: I don't think so, and I'd be really surprised
< wumpus>
the only use of protobuf is the payment request code in bitcoin-qt and that hasn't seen serious changes since... forever
< wumpus>
that looks like you need to make clean (or even clear your tree) and rebuild -- this is most likely caused by the protobuf version on your system changing through package managers or such
< jamesob>
thanks, I'll give that a shot
< wumpus>
(pb.h isn't part of the repository but generated)
< quitobro>
hey guys i have a question about best practice for managing watch-only addresses
< quitobro>
specifically, which RPC commands are best to use in order to *get transactions tied to an input address*
< quitobro>
i was expecting `getreceivedbyaddress` to take an address param and return a list of UTXOs or tx's
< quitobro>
furthermore, when i was exploring the `*byaccount` RPC commands it seems like many of them are deprecated - is the 'account' concept being deprecated or just select commands like `getreceivedbyaccount`?
< sipa>
quitobro: the accounts concept is deprecated
< sipa>
quitobro: if you import a watch only address, the usual RPCs like getreceivedbyaddress will work
< sipa>
but you first need to import them
< sipa>
quitobro: labels will replace accounts; so you'll still be able to give a label to an address and see transactions to a label etc
< sipa>
but there won't be a "label balance" like there is an account balance
< sipa>
they're just a way to tag addresses for receives
< quitobro>
sipa: ok thanks. yea, `getreceivedbyaddress` is nice but i want something which returns the transaction history at address, not just the balance...
< sipa>
quitobro: listreceivedbyaddress
< quitobro>
i suppose i can go thru `listtransactions` and filter by address?
< quitobro>
ah, ok, and then filter thru the list of tx summaries grabbing only the ones i care about, or something like that. thank you sipa!
< quitobro>
this is what i needed :)
< jamesob>
wumpus: turns out I had a mismatch between my protoc version (newer) and libprotobuf-dev
< wumpus>
jamesob: whoops
< quitobro>
sipa: one more question - when calling `importaddress` with `rescan=true`, will that operation be blocking other reads/writes to the local blockchain database?
< sipa>
quitobro: it will block pretty much everything
< quitobro>
sipa: okay sounds like we may need to schedule our rescans very carefully then
< sipa>
quitobro: you should import addresses before they're being used :)
< quitobro>
how long do rescans take usually? relative to say initial sync time?
< sipa>
a significant fraction
< gmaxwell>
quitobro: Can you talk about your usecase? why are you performing rescans at all?
< quitobro>
hm okay... for our application we don't manage users' keys, but want to provide essentially a blockchain explorer... is there a way to get tx & balance details for an arbitrary valid address, without importing as watch-only?
< quitobro>
we looked at using a 3rd party service but we don't want those blockchain queries to potentially reveal customers' addresses
< quitobro>
and would prefer to run our own nodes and fulfill those requests
< quitobro>
we sell a cryptocurrency trading platform as a software service
< quitobro>
so there is a large blockchain explorer-esque component as part of e.g. trade settlement
< gmaxwell>
Importing as watching before issing the addresses is the canonical, supported way. Your next best alternative is to write your own indexer. The issue is that the resource costs of indexing all transactions in history will continue to rapidly grow... so most people who setup that way will eventually need to switch to using a centeralized service due to the resource costs.
< gmaxwell>
There hasn't been much interest in maintaining that kind of index in bitcoind because of the above.
< quitobro>
gmaxwell: in other words everyone's index needs/requirements vary so there isn't a good way to implement arbitrary indexes in bitcoind?
< quitobro>
or rather indexes for arbitrary watch-only addresses
< gmaxwell>
We may have a feature in 0.18 that makes rescan much faster (e.g. a minute maybe, instead of hours), but likely still too slow for your application.
< quitobro>
gmaxwell: cool what is that feature? sounds quite nite
< gmaxwell>
quitobro: diversity of needs is one issue, but scalablity is another. At least in the community people aren't generally excited on working on technology that will only be usable on large dedicated servers, if not now, then in a couple years. If someone showed up and wanted to add optional indexing, and was willing to jump through the right hoops to isolate it, I think the contribution would be
< gmaxwell>
welcome.
< quitobro>
nice*
< gmaxwell>
quitobro: use of BIP-158 filters locally. Basically for every block we'd save a small fingerprint of the addresses involved in the block.
< gmaxwell>
Then scanning only has to check those, and not the whole blockchain.
< gmaxwell>
it's still a linear scan, rather than an index, but of a lot less data.
< gmaxwell>
There is a pull req implementing the filters, but not the wallet rescan using it, yet.
< gmaxwell>
It's too late to make 0.17 right now, but I think it's somewhat likely for 0.18.
< quitobro>
gmaxwell: i see, ok maybe if we continue down this path (as opposed to running a super beefy, indexed dedicated server) we will be able to make a contribution
< gmaxwell>
If nothing else, being willing to show up and test it would help.
< quitobro>
definitely; should we just keep our eyes on bitcoin-dev-mailing-list?
< gmaxwell>
In general, when there are features that are mostly interesting for commerical players, we'd still welcome them, but industry needs to step up and do more of that work... there is just too much to do. :)
< gmaxwell>
After it's merged presumably there will be more PRs to actually make use of it.
< midnightmagic>
I actually thought working on the weird higher-end functionality was the best thing I used to do professionally. (as limited as that was)
< MarcoFalke>
Just a PSA for member of the GitHub-labels group:
< MarcoFalke>
You can tag with "Needs gitian build" and DrahtBot will create a built in the next day or so
< promag>
provoostenator: let me know if you can try #13791 in windows? ty!