< bitcoin-git>
[bitcoin] pstratem opened pull request #15597: Generate log entry when blocks messages are received unexpectedly. (master...2019-03-12-net-unexpected-block) https://github.com/bitcoin/bitcoin/pull/15597
< gmaxwell>
phantomcircuit: good spotting
< Sentineo>
MarcoFalke: did not understand you finding about those gpg keys. Is the issue with the key itself or something else? I did not open an issue yet, I can if it is just not my noobness what causes it not working :)
< gmaxwell>
Sentineo: run gpg --recv-key AC6626172E00A82CFFAE8972A636E97631F767E0
< midnightmagic>
w/w 3
< Sentineo>
gmaxwell: thanks, that seemed to help. As it is still working, it took about a minute for it to fail before. I wonder where you got the key ID from though. Mind explaining so I can fix the issue later myself if happens again?
< bitcoin-git>
[bitcoin] luke-jr opened pull request #15600: lockedpool: When possible, use madvise to avoid including sensitive information in core dumps or forked process memory spaces (master...lockedpool_dontdump) https://github.com/bitcoin/bitcoin/pull/15600
< provoostenator>
Does the MSVC build ignore configure.ac?
< bitcoin-git>
bitcoin/0.18 a01925c Wladimir J. van der Laan: doc: Pre-rc2 translations update
< wumpus>
provoostenator: I think so, yes
< provoostenator>
Actually I don't think it's ignored. It's just that .appveyor.yml doesn't bust the cache when it changes. Fixed as part of #15457 (hopefully)
< jonasschnelli>
(if we are fine with adding crypto primitives)
< wumpus>
jonasschnelli: yes
< wumpus>
at least if there's concept ACK agreement to add it
< jonasschnelli>
Okay... added to the list. So waiting for Concept reviews
< wumpus>
other topics?
< sipa>
instagibbs had a topic
< gmaxwell>
RC2 is tagged. hurrah
< instagibbs>
any "needs" for 0.18 being the topic, in other words
< wumpus>
#topic 0.18.0rc2
< wumpus>
it's not tagged yet but intend to do so after the meeting, if no one complains
< gmaxwell>
oh. :P
< gmaxwell>
Well great, that would be a good time for people to start building too, I guess?
< instagibbs>
as an aside I think HWI is near 1.0, 1 open issue right achow101 ?
< sipa>
i'm a bit surprised by the amount of people (claiming to) use the reject message on the ML... should we proceed with making it default off in 0.18?
< achow101>
instagibbs: yeah, just one thing needs review
< wumpus>
#topic reject message default
< wumpus>
sipa: I've alwys been divided on removing it to be honest
< sipa>
i would very much like to disappear, but my assumption was that it was much less used than people are claiming now
< gmaxwell>
The comments on the list have, once you sort through the speculative claims, all been "I use it for debugging"
< wumpus>
that makes sense, that's what it's for
< gmaxwell>
I'm still confused by those however, -- because is it also "I want to debug a lite wallet without running a full node myself?"
< wumpus>
then again disabling it by default wouldn't affect that use case, just make sure it's enabled on the node used for debugging
< gmaxwell>
Right.
< instagibbs>
right I don't really get it.
< instagibbs>
have a full node for debugging?!?
< gmaxwell>
I think its unfortunate that that this has come up in the context of removing it entirely.
< achow101>
gmaxwell: istm it's debugging user submitted bug reports, not the developer debugging his own app
< sipa>
yeah that's the case in the schildbach wallet
< promag>
hi
< MarcoFalke>
the neutrino wallet is using them as well, apparently
< gmaxwell>
achow101: Yes, but capturing the transaction itself would seem to be strictly superior for that.
< gmaxwell>
MarcoFalke: along with a bunch of other assumptions about trusting the nodes its communicating with.
< gmaxwell>
Which is already a security model we've made a conscious decision to not add support for.
< gmaxwell>
MarcoFalke: and clearly not using them with existing nodes, since they require other protocol extensions.
< achow101>
gmaxwell: certainly. but at the same time, having the reject message can provide better ux as the user can see why their transaction was rejected (in theory)
< MarcoFalke>
They way one wallet makes it trustworthy is to send the tx to all nodes and see if at least half of them report the same reject reason
< gmaxwell>
MarcoFalke: that doesn't make it trustworthy.
< MarcoFalke>
Yeah, I am not going to start to explain what is wrong with that, even
< gmaxwell>
Right...
< sipa>
i'm just bringing it up because i think this ML discussion should have happened before making it disabled by default, rather than before removing it entirely
< MarcoFalke>
sipa: You are right
< MarcoFalke>
It is sort of pointless to make them disabled by default, but then not remove them
< moneyball>
agree with sipa
< MarcoFalke>
So they should probably be enabled by default again for now. Not sure if that makes it into 0.18.0
< jnewbery>
it's not too late to revert #13134
< gribble>
https://github.com/bitcoin/bitcoin/issues/13134 | net: Add option `-enablebip61` to configure sending of BIP61 notifications by laanwj · Pull Request #13134 · bitcoin/bitcoin · GitHub
< jnewbery>
or just change the default to 0
< gmaxwell>
It was covered in the optech newsletter, but I don't see a prior ML discussion of it.
< MarcoFalke>
yeah, just change the default
< MarcoFalke>
No need to remove the option
< gmaxwell>
MarcoFalke: it's not pointless to disable them by default but not remove them.
< gmaxwell>
MarcoFalke: they're a source of a non-trivial amount of junk protocol chatter.
< gmaxwell>
MarcoFalke: and the reliance on them creates its own problems.
< gmaxwell>
(including incentives to sybil attack the network)
< MarcoFalke>
In light of the spv wallets that use them, default by off is the same as having them removed
< sipa>
ideally we can convince people to switch to more reliable methods before disabling support for it
< gmaxwell>
I don't see anything terrible about delaying disabling it. But I am also not sure what is going to change.
< gmaxwell>
sipa: there isn't anything to switch.
< gmaxwell>
sipa: the two respondands saying they're using it (1) only log it, (2) don't work against bitcoin core already.
< sipa>
gmaxwell: ?
< gmaxwell>
sipa: android wallet doesn't actually do anything with the messages but log them, the neutrino wallet requires protocol extensions that are only in btcd.
< sipa>
right, for now
< gmaxwell>
so whats there to switch?
< sipa>
ou suggested logging the tx itself instead of the reject message, for example
< gmaxwell>
Okay, fair point!
< sipa>
plus there are other methods like seeing transactions announced back to yourself (okay, not very reliable either)
< sipa>
i just wish this discussion was more resolved before we possibly anger people needlessly by ripping something out they stubbornly rely on now
< gmaxwell>
sipa: one client does that but sends their txn to all peers :) ...
< gmaxwell>
12:20:11 < gmaxwell> I don't see anything terrible about delaying disabling it. But I am also not sure what is going to change.
< jnewbery>
sounds like the action item is to revert #14054 before 0.18
< sipa>
yeah, it's a fair point that delaying may just literally mean delaying it
< sipa>
but perhaps it does resolve the discussion further
< gmaxwell>
I think we should probably make it clear that we still intend to disable it by default.
< MarcoFalke>
jnewbery: I am fine with that, but I hope that doesn't give a signal that they are encouraged
< MarcoFalke>
They should still go in the long term
< sipa>
agree
< gmaxwell>
I mean I think it was clear years ago that these shouldn't be used the way people are advocating using them... like on day one.
< sipa>
i doubt that was clear to everyone :)
< cfields>
Hasn't our unwritten policy always been to deprecate in one release, and non-default/remove in the next?
< MarcoFalke>
cfields: I tried to do that, but apparently people only picked up on this when I wanted to remove them
< sipa>
cfields: for RPC that works... for P2P services it's harder
< jnewbery>
That's easier for RPCs because deprecation is immediately obvious to the user
< gmaxwell>
some of the things people are reporting doing with them haven't worked right for a long time (or ever)... (like trying to differentiate between already spent and other causes of invalidity)
< moneyball>
It would also help to better educate what they should do as an alternative. My reading of the thread still has me uncertain, and they also do not seem to know what to do.
< gmaxwell>
Actually disabling it in a release has that effect somewhat, as it'll be years before it's gone completely on the network.
< cfields>
sipa: I just meant that if it's going to be non-defaulted in one version, seems only fair to put it through a deprecation cycle in the preceding one.
< jnewbery>
how do you deprecate a P2P message in a way that users notice?
< moneyball>
Good next steps IMO would be 1) revert in v0.18 2) communicate this to the list but also emphasize the long-term plan 3) educate on alternatives for these users
< sipa>
cfields: right, but what does 'deprecation' mean here? if it's not disabled by default, it's not even observable
< gwillen>
cfields: but I think sipa's point is that deprecation doesn't notify the people who care
< gmaxwell>
moneyball: to some extent what wants to be done just can't be done, even with rejects. You can't find out if the network is taking your transaction, it's just not something anyone can tell.
< cfields>
Just in the release notes.. we intend to flip this to disabled-by-default in the next release.
< sipa>
cfields: sgtm
< cfields>
that's all I meant by "deprecate".
< gmaxwell>
cfields: ack.
< wumpus>
so I guess this does hold up rc2 tagging then
< jnewbery>
I'm happy to open that PR now, unless MarcoFalke wants it
< MarcoFalke>
jnewbery: go for it
< gmaxwell>
jnewbery: go do it.
< jnewbery>
I'll do it today
< provoostenator>
You can make it noticable by always returning reject :-)
< gmaxwell>
lol
< gmaxwell>
provoostenator: nope, in fact.
< sipa>
reject 00 "The operation completed succesfully"
< gmaxwell>
provoostenator: because reject isn't doing anything in existing software except ending up burried in some logs.
< achow101>
what's the long term plan for removal? default disable in 0.19, remove in 0.20?
< gwillen>
I think starting mailing list discussion of what the alternatives are, and getting moneyball to spread it in the newsletter, is probably the best way to spread the good word :-)
< MarcoFalke>
achow101: Something like that
< MarcoFalke>
[15:26] <moneyball> Good next steps IMO would be 1) revert in v0.18 2) communicate this to the list but also emphasize the long-term plan 3) educate on alternatives for these users
< gmaxwell>
gwillen: right, what I would recommend for these logging cases is that they log the transaction and current block height.
< MarcoFalke>
gmaxwell: you mean the full tx in hex?
< gmaxwell>
MarcoFalke: whatever is required for them to be able to reproduce! in some applications it might be less than the full tx.
< gmaxwell>
cfields: you're misunderstanding, the wallet should log its own transaction.
< MarcoFalke>
Yeah, its only the own wallets txs
< cfields>
gmaxwell: I was, thanks.
< gmaxwell>
MarcoFalke: e.g. if the wallet always stores the tx like bitcoin core, the current block and txid are enough.
< gmaxwell>
cfields: basically the debugging case is "User says my transaction was never confirming" ... what do you do? Andreas is pointing out that reject messages have been helpful.
< gmaxwell>
Another thing is that I see no feefilter support in that software... so obviously processing feefilters would allow them to locally learn about some of the failure cases they care about.
< gmaxwell>
right now reject basically only tells you didn't meet minfee vs "something else happened at the first hop", feefilter + listen to hear it back gets you essentially that, but more reliable.
< sipa>
gmaxwell: though feefilter can't be relied upon either in an adverserial setting
< sipa>
it is less DoS risk though
< gmaxwell>
sipa: no but it's probably better than reject messages.
< gmaxwell>
esp if you obey it, e.g. don't send a txn that would violate the filter to a particular peer.
< sipa>
end topic?
< gmaxwell>
ack
< wumpus>
other topics?
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Mar 14 19:40:57 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< gmaxwell>
Can someone explain this tweet people were passing around? https://twitter.com/pierre_rochard/status/1104785795523719169 I don't understand how fullblock spv mode and the BIP157 related PRs are at all compariable/substutiable for each other.
< gmaxwell>
(someone in #bitcoin tried asking me my opinion about that debate earlier, and I thought it would be nice to actually understand it before responding...)
< bitcoin-git>
[bitcoin] jnewbery opened pull request #15602: [p2] Enable reject messages by default (0.18...reject_message_by_default) https://github.com/bitcoin/bitcoin/pull/15602
< moneyball>
My understanding is that pierre_rochard is focused on onboarding new Bitcoin users via Lightning (with his Lightning Powered Users), and he would like as many of them as possible to run full nodes, but he wants them to be able to use Bitcoin immediately so wants to support BIP157 style light clients. He's also saying if Core doesn't merge support for BIP157, he'd maintain a version of Core with it merged, and run
< moneyball>
it himself to support his users
< harding>
gmaxwell: pierre_rochard maintains an installer that installs Bitcoin Core, LND, and a LN wallet that's capable of using BIP157/158. I think his desire is to allow people to immediately start using LND and the LN wallet using BIP157 filters served from his node while their Bitcoin Core node syncs. That is, I don't think he's talking about hybrid SPV in Bitcoin Core by hybrid SPV via LND/Neutrino/some other wallet.
< harding>
s/by hybrid/but hybrid/
< gmaxwell>
harding: for that sort of thing, just fetching the blocks seems completely reasonable even ideal..
< provoostenator>
harding: that seems a plausible interpretation
< harding>
gmaxwell: I agree.
< gmaxwell>
So then I guess the confusion there would be that what he'd really want as the alternative is 'hybrid spv mode' in the ln wallet? not in bitcoin core.
< sipa>
only partially related, i think there is a lot of confusion about what "bip157" means; there is (a) the spec, allowing software to implement the filters in a private protocol like wasabi does (b) support for it in bitcoin core via RPC (what the current PRs do) (c) exposing it in core and other software via P2P for trusting peers to use (d) exposing it in core via P2P for non-trusting peers (e) a
< sipa>
softfork to prevent non-trusting peers from...
< sipa>
lying about filters (assuming trust in hashrate majority)
< gmaxwell>
indeed.
< sipa>
there seems to be a lot of antagonism on twitter against (d)
< gmaxwell>
I'm not really aware of the twitter stuff (other than having been given that link) ... but my thought for many months is that I'm super excited about having the filters to make rescans usable again... and super concerned about them starting a new wave of bip37 like wallets that just blindly trust things.
< harding>
gmaxwell: AFAIU, he just wants some way for people to start using an LN wallet in the SPV trust model while their node syncs. I'm not sure he cares how it happens. I myself don't know why BIP157/158 is entangled in this, except that he might think it's necessary to accomplish that.
< echeveria>
sipa: (f) using it for wallet rescans.
< sipa>
echeveria: good point; i'd put it under (b) as trust model, but agree
< echeveria>
(f) ii. using it for wallet rescans while pruned.
< gmaxwell>
harding: yea okay, I'd even say BIP157/158 is a pretty weak way to accomplish that particular case. ... deploying a new protocol would take a lot of time in the best case, while just fetching the blocks works now against the existing network.
< harding>
I've been trying to use "BIP157" for the filters themselves and "BIP158" for the P2P parts, but it's not always that clearcut.
< sipa>
harding: it's the other way around :p
< gmaxwell>
Fetchign the blocks has better security/reliablity properties. It uses more bandwidth, but that doesn't seem terribly important in the "while the user's node syncs" case.
< harding>
gmaxwell: yeah, and any client that supports BIP157 must, by necessity, also support grabbing and parsing full blocks anyway, so supporting grabbing all blocks after a certain height ought to be a trivial addition.
< harding>
sipa: is it really? /me facepalms
< gmaxwell>
the bandwidth usage isn't even all that great if you're also just talking about now-forward and not the history. (14kbit/sec whoppie)
< sipa>
harding: yeah 158 defines the filters, 157 is the p2p protocol to expose it
< gmaxwell>
echeveria: (f) ii. -- thats cute.
< gmaxwell>
being able to rescan with a pruned wallet would be pretty nice... though carring around the filters is a pretty big hit on top of pruning (2x disk space usage maybe?)
< gmaxwell>
probably worth it, just because a lot more users could prune, which is becoming increasingly critical as the history is now getting larger than the most popular SSD sizes.
< sipa>
iirc the filters are around 20 KiB per block?
< gmaxwell>
they're smaller for earlier blocks, but not as much smaller as the blocks are.
< echeveria>
gmaxwell: one PR has a comment that it's about 10GB today.
< gmaxwell>
ouch, a bit worse than I thought.
< echeveria>
so pruned would be 10GB + 3GB UTXO + 0.6GB blocks, but allows wallet rescan.
< gmaxwell>
yeah....
< instagibbs>
this is where commitments would help i guess
< gmaxwell>
instagibbs: I don't think so, actually.
< instagibbs>
oh?
< echeveria>
unpruned would be 10GB + 220GB, so a 5% hit for significantly faster imports.
< gmaxwell>
I mean, having to fetch 10GB+growing of data every time you want to rescan isn't really "recan supported" :P
< instagibbs>
oh hah right
< instagibbs>
well, you could forget filters on a rolling basis
< instagibbs>
but i guess the same logic applies
< instagibbs>
for just redownloading them :)
< gmaxwell>
yes, though we've not yet really figured out how to make time windowed rescan work well for people.
< echeveria>
gmaxwell: sipa: looks better with 3.8GB. same size as the UTXO give or take.
< gmaxwell>
echeveria: yeah, for the moment.. 2x. not terrible.
< roasbeef>
i think in that tweet pierre is just saying that if bitcoind doesn't get fully p2p support eventually, he'd maintain a fork that implemented it, as earlier in the thread ppl proposed full block download as an alternative
< pierre_rochard>
gmaxwell: "for that sort of thing, just fetching the blocks seems completely reasonable even ideal.." agree, and c-lightning does that with spruned. This would work for LND as well, but ZMQ emulation needs to be added to spruned. I would get busy working on that, but I hear LND mainnet neutrino + btcd isn't far off.
< pierre_rochard>
My activism for serving up filters with bitcoind is because I'd rather be running bitcoind than btcd, just for historical reasons
< pierre_rochard>
"I think his desire is to allow people to immediately start using LND and the LN wallet using BIP157 filters served from his node while their Bitcoin Core node syncs."
< pierre_rochard>
This is exactly right
< pierre_rochard>
people who have to wait hours/days for IBD to be able to use LND often end up just taking the shortcut to a custodial lightning wallet. We can have both rapid onboarding and full nodes if we use a light client as a stopgap
< MarcoFalke>
rc2 ready to tag?
< achow101>
provoostenator: what tests do you have that require importing private keys into the keypool?
< achow101>
I would rather not add such a feature just so a test can work easier
< webuser3254>
hello, I have a question regarding pruned mode and importprivkey. I have just imported hundreds of privkeys from my electrum wallet to bitcoin gold core pruned wallet but I can't see any addresses or balances. I can't seem to be able to do rescan on pruned mode. Is there a way to see my balance or move ALL the coins to a new address?
< webuser3254>
I appologise for posting it here but I can't do it #bitcoin for some reason :/
< echeveria>
what reason? the channel doesn't require registration.
< webuser3254>
"Cannot send to nick/channel: #bitcoin"
< jimpo>
sipa: The line between BIP 157 and 158 is kind of muddied, but the notion of filter types and a chain of headers is in 157. 158 is just the specification of the construction of the basic filter type.
< jimpo>
Since the index supports multiple filter types (ish), I feel like it's more related to 157 than 158
< sipa>
jimpo: fair
< sipa>
still, the actual filter is in 158
< jimpo>
Eh, the data type corresponding to the BlockFilter struct is in 157 (the cfilter response)
< achow101>
webuser3254: rescanning requires the full blockchain. if you are pruned, you don't have the full blockchain so you can't rescan. you could use scantxoutset to get all of the utxos for our private keys and then create a raw transaction using those
< jimpo>
and the index PR only never does any encoding/decoding of the filter byte vector
< jimpo>
but I agree it's kind of in the middle of the two
< sipa>
yeah
< webuser3254>
achow101, thank you but somehow craeting raw transaction sounds complicated to me right now... my bad again for spamming this channel.
< webuser3254>
thanks for all the work guys!
< bitcoin-git>
[bitcoin] jnewbery opened pull request #15604: [docs] release note for disabling reject messages by default (master...release_notes_bip61) https://github.com/bitcoin/bitcoin/pull/15604