< bitcoin-git>
[bitcoin] Empact opened pull request #13560: Drop unused serialization support from CAddresss (master...caddress-serialization) https://github.com/bitcoin/bitcoin/pull/13560
< provoostenator>
I'm getting crashes during IBD of a pruned node on one of my ARM devices. It tends to happen during later prune events (e.g. 2015). I set a fairly high dbcache, but during those events dbcache is << RAM (and there's a swap).
< provoostenator>
Worse, it sometimes tells me to reindex.
< provoostenator>
Log shows no indication of why it crashed, but my SSH connection to the machine tends to die when it happens, which does suggest OOM?
< provoostenator>
Perhaps it's just crappy hardware, but given the weird behavior with pruned nodes and large dbcache in IBD I saw in #12404, might indicate some subtle bug.
< bitcoin-git>
[bitcoin] Empact closed pull request #13560: WIP: Drop unused serialization support from CAddresss (master...caddress-serialization) https://github.com/bitcoin/bitcoin/pull/13560
< bitcoin-git>
[bitcoin] droark opened pull request #13561: Qt: Remove unnecessary image buffer for Mac dock icon (master...rm_icon_buffer) https://github.com/bitcoin/bitcoin/pull/13561
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #13562: travis: Switch back to trusty for now (master...Mf1806-qaTravisTrusty) https://github.com/bitcoin/bitcoin/pull/13562
< bitcoin-git>
bitcoin/master ea49e06 practicalswift: tests: Fix incorrect documentation for test case cuckoocache_hit_rate_ok
< bitcoin-git>
bitcoin/master c93c360 MarcoFalke: Merge #13551: tests: Fix incorrect documentation for test case cuckoocache_hit_rate_ok...
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #13551: tests: Fix incorrect documentation for test case cuckoocache_hit_rate_ok (master...truth-in-advertising) https://github.com/bitcoin/bitcoin/pull/13551
< promag>
qt: funny thing, connect(..., &Foo::foo) doesn't work if foo is a private slot, BUT connect(..., SLOT(foo()) works..
< bedo>
Hi all, it's only me or the testnet are generating crazy amount of block?
< jonasschnelli>
bedo: yes. Someone mining with an ASIC farm I guess
< jonasschnelli>
4-5 seconds interval between blocks. :)
< bedo>
jonasschnelli: today is hard day for develop over bitcoin
< jonasschnelli>
bedo: use regtest. :)
< Lightblock_>
Hi
< Lightblock_>
happy to be here
< Lightblock_>
sorry if silly question, could anyone please suggest a proper way to the bitcoin transaction ID given that I know the blockheight txindex and outputindex ?
< Lightblock_>
sorry, seems like I have put in the worng channel
< Lightblock_>
nevermind, asked in the #bitcoin already
< bedo>
jonasschnelli: yep, is the only way, will see you at BOB?
< jonasschnelli>
bedo: Yes. Sure!
< bedo>
jonasschnelli: perfect ;)
< bitcoin-git>
[bitcoin] jnewbery opened pull request #13564: [wallet] loadwallet shouldn't create new wallets. (master...dont_load_nonexistent_wallet) https://github.com/bitcoin/bitcoin/pull/13564
< bitcoin-git>
[bitcoin] Empact opened pull request #13565: Fix AreInputsStandard test to reference the proper scriptPubKey (master...p2sh-tests-pub-key) https://github.com/bitcoin/bitcoin/pull/13565
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu Jun 28 19:00:56 2018 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< jnewbery>
half a hi. May be a little distracted for the next ~45 minutes
< wumpus>
I've had a really crappy week so haven't been able to do much, sorry for that
< sipa>
sorry to hear that
< wumpus>
#topic high priority for review
< sipa>
Currently on the list: #13425 #12196 #13062
< gribble>
https://github.com/bitcoin/bitcoin/issues/13425 | Moving final scriptSig construction from CombineSignatures to ProduceSignature (PSBT signer logic) by achow101 · Pull Request #13425 · bitcoin/bitcoin · GitHub
< wumpus>
jonasschnelli: I think it makes sense to merge something, as you say it has a lot of ACKs, further improvements can be done later unless the current state is really unacceptable
< sipa>
i also have a prototype implementation for most of it
< wumpus>
nice!
< jonasschnelli>
Yes. I really like it.
< promag>
wumpus: for 0.17, dyn multi-wallet in the UI is required?
< sipa>
it is a general language that encodes all information about how to spend a whole set of keys with associated addresses/scripts/private keys/.... into one string, including support for multisig etc
< wumpus>
promag: why?
< promag>
it's a question
< wumpus>
promag: we want toh ave things consistent before a release, at least, apart from that it's simply a matter of what makes it in
< sipa>
so import/export would operate at the level of those descriptors, instead of individual keys/scripts/pubkeys/hdchains/...
< sipa>
importmulti is already compatible with that design, for a large extent
< sipa>
the entirety of that idea is certainly not for 0.17, however
< sipa>
but that doesn't mean it can't be used already in relatively small scoped things already
< sipa>
and scanutxoset is one of those
< jonasschnelli>
what API changes would you propose for scantxoutset so we can migrate towards the output descriptors in the same cycle as migrating importmulti?
< wumpus>
that would be very last minute, but at least using it as a guideline to be compatible with the current stuff makes sense
< sipa>
jonasschnelli: you may not like this, but what about just dropping xpub support from the PR right now
< jonasschnelli>
sipa: this makes the PR pretty useless... :(
< sipa>
and then i'll PR the descriptor language, together with integration into scanutxoset
< sipa>
jonasschnelli: i understand
< sipa>
feel free to disagree
< wumpus>
it makes sense to divide it up like that
< jonasschnelli>
But if the API break would be complex and painful, we can do that.
< wumpus>
makes tha change smaller and less complex
< jonasschnelli>
I don't disagree... :)
< wumpus>
(besides sipa's point of course)
< sipa>
if your concern is that it may not make it in for 0.17, you can still PR the (already written) xpub support as is later, before feature freeze?
< jonasschnelli>
Sure... I guess its also not utterly bad if the xpub will be in 0.18.
< jonasschnelli>
Okay. Will remove the xpub stuff
< sipa>
thank you. i promise i'll work on having a PRable implementation soon
< jonasschnelli>
The question of a gap limit came up recently (related to the xpub situation) but this concept seems not to work with utxo based scans..
< jonasschnelli>
So a fixed lookup window makes more sense IMO
< sipa>
agree
< sipa>
jonasschnelli: that's actually a good point against having a gap limit inside the descriptor language
< sipa>
(as a gap limit is not relevant for all use cases)
< jonasschnelli>
gap limit is a broken concept IMO
< jonasschnelli>
I would not use it in the descriptors...
< sipa>
in the context of high priority PRs that's all i have to say about it; but we can discuss this idea in more detail as a separate topic if there's interest
< jonasschnelli>
Thanks for working on this sipa. will give more feedback soon.
< wumpus>
any proposals for adding high-priority PRs, or rmaoving them?
< wumpus>
heh I already considered doing a #topic change
< jonasschnelli>
I have two topic requests: a) Cipherseed, b) Cores BIP32 derivation "standard"
< promag>
wumpus: I'll complete #13100 soon and it could go to hp list
< jonasschnelli>
I have a specification draft for a new seed format similar to BIP39 with some neat properties and – before sending to the ML – would appreciate feedback.
< luke-jr>
(if there were a BIP, I would think it should cover the whole wallet format, not *just* derivation)
< sipa>
luke-jr: saw my descriptor proposal above? :)
< achow101>
just documnting the derivation in the docs repo is sufficient imo
< jonasschnelli>
I think following BIP32 for "hot" wallets with private key export options is not ideal... Electrum does that as example
< sipa>
my point is that i don't think our scheme is particularly an improvement over alternatives, or has all that much design we want to convince others about
< sipa>
it's just one of many choices, and the one we made
< jonasschnelli>
Agree with that. Yes.
< sipa>
so we should just document it
< jonasschnelli>
ack
< achow101>
+1
< gmaxwell>
Seems good.
< wumpus>
agree
< wumpus>
I think this leaves sipa's topic, but I think that's more or less discussed already?
< sipa>
yeah, probably needs people reading the idea first to discuss more; can be done offline
< wumpus>
right
< jonasschnelli>
sipa: how would it interact with the keypool, flexible keypath?
< jonasschnelli>
and a xpub
< sipa>
jonasschnelli: keypool goes away
< wumpus>
good riddance
< sipa>
there is ephemeral data in the wallet associated with the descriptor (which is a black box, and descriptor specific), but in practice contains the expanded pubkeys
< sipa>
that takes the place of the keypool- but those things don't all translate to independent keys in the wallet
< sipa>
there would just be a single private key in your wallet, for example
< sipa>
(or none at all; it can be in a hardware device too)
< sipa>
flexible keypath... the descriptor just contains the path
< sipa>
you can change it to whatever you like (but default wallets would of course pick some standard scheme)
< sipa>
or rather you can import things with whatever path you like
< wumpus>
makes sense
< jonasschnelli>
Would it make sense that the descriptor support pkh(d34db33f/44'/0'/0':<seed>/1/i). (seed along with xpriv)?
< jonasschnelli>
for backward compatibility
< sipa>
jonasschnelli: i've thought about that, but that makes descriptors non-canonical
< sipa>
(as in: you can't convert them to "public" form and back, and retain all information)
< sipa>
i'm unsure how to deal with that; my thinking is initially no
< sipa>
you can always implement it as an extra utility that converts from seed based format
< jonasschnelli>
There is always the option of externally converting the seed to an xpriv, yes
< jonasschnelli>
we can encode seeds into xprivs *duck*
< gmaxwell>
Not to hijack, but has there been any progress towards implementing P2P link ephemeral encryption lately? I know we were kinda waiting for some other networking refactors.
< sipa>
cfields: ping?
< wumpus>
#topic P2Plink ephemeral encryptio
< jonasschnelli>
I'm ready to pick that up any moment but was under the impression that sipa had plans to implement it
< jonasschnelli>
I started with the implementation but halted at some point...
< jonasschnelli>
I'm also not sure if we should delay it further more for "refactors"
< gmaxwell>
I believed sipa did too, but asformentioned delays.
< cfields>
heh, I was waiting on it to firm up. Guess we were waiting in circles :)
< wumpus>
hehe
< cfields>
jonasschnelli: for sure
< jonasschnelli>
BTW: armory has implemented it and has plans to PR it to Core
< gmaxwell>
Sipa and I made some major advances in the authentication part but the encryption doesn't need to wait on that.
< jonasschnelli>
(not sure how soon and in what quality)
< sipa>
cfields: waiting for encryption proposal to firm up before implementing it? or before continuing with network refactors?
< wumpus>
jonasschnelli: oh wow
< jonasschnelli>
Agree with gmaxwell. Authentication can be added later.
< cfields>
sipa: i've had to put the net stuff on the backburner for now, so certainly don't wait for that.
< sipa>
cfields: cool
< jonasschnelli>
cfields: I think BIP151 is almost final (there is some issues with the version handshake)... the only thing that was holding me back where possible network refactors to first wait for
< cfields>
I'm happy to help with the implementation. I was thinking we were waiting on the auth stuff.
< luke-jr>
jonasschnelli: it can't be Final until it is adopted..
< gmaxwell>
no, they're designed to operated independantly.
< jonasschnelli>
Auth is additional and implementation wise it comes after 151
< sipa>
we can implement 151 without 150
< gmaxwell>
I would rather not use the prior auth design, we have better ones now.
< jonasschnelli>
Yes. 150 can also be replaced (coexist) with other auth proposals
< sipa>
the other link is just some cool trick, not a serious proposal
< jonasschnelli>
Okay. If no one else wants to work on the implementation, I will continue then with BIP151 impl.
< gmaxwell>
Basically there was an open question of if we wanted the encryption handshake to operate in such a way that there are no fixed bytes for easy blocking/detection. But I think we thought the benefits were too dubious.
< gmaxwell>
Esp since traffic patterns will identify bitcoin p2p links very clearly.
< cfields>
too dubious? you mean foiled by dpi anyway?
< gmaxwell>
And so probably better to just stick to something simple.
< jonasschnelli>
Agree
< wumpus>
hiding what kind of connection something is is very difficult
< gmaxwell>
cfields: foiled by traffic analysis or smarer DPI (that does EC operations to match traffic)
< gmaxwell>
smarter*
< gmaxwell>
People can always carry bitcoin over other transports in any case, ... ones that can do things like pad out to a constant bitrate...
< gmaxwell>
but we're certantly not going to make BIP151 do that. :P
< wumpus>
might be creeping the scope a bit too much
< gmaxwell>
in any case, changing the handshake to be harder to detect was the only 'maybe' design change that I'm aware of any of us considering.
< gmaxwell>
For 151.
< jonasschnelli>
You mean an obfuscation of the encryption handshake?
< gmaxwell>
So I think we're good to implement, and the only changes that might be proposed would be ones that arose as a side effect of implementing and benchmarking.
< gmaxwell>
jonasschnelli: yes.
< jonasschnelli>
Yes. I think there is freedom to change the specs during implementation...
< gmaxwell>
And my view is that it's not worthwhile because without other more complex obfscuation (which will be bandwidth costly) it'll still be pretty detectable.
< jonasschnelli>
It's not really deployed on the network yet
< gmaxwell>
right.
< jonasschnelli>
Yes. Better not obscure and put efforts in a long term solutions (stuff like the ScrambleSuit)
< cfields>
my only complaint was that it required message parsing to complete the handshake, but it's been a while since I looked, so I'm not sure if that's still the case. I also got the impression that nobody else seemed all that bothered by that anyway.
< jonasschnelli>
can you elaborate a bit more on " it required message parsing to complete the handshak"?
< cfields>
jonasschnelli: we can discuss after the meeting, I need to take a look at the current spec
< jonasschnelli>
sure. Thanks cfields
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Jun 28 19:59:47 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< cfields>
jonasschnelli: If the first 32bytes over the wire are a pubkey, the network layer can do the handshake and notify us of a new connection only after it's done. The message processor wouldn't have to know or care about encryption, it just pushes messages, and the network layer handles the rest automatically.
< cfields>
If, however, messages have to be parsed before doing the handshake, the net layer has to be told to switch to encryption, which imo is awkward.
< jonasschnelli>
Yes. I think your right...
< jonasschnelli>
We should probably do a dummy handshake in advance... what would you think about that cfields?
< cfields>
(I'm assuming that encryption would be handled at the net layer, I suppose doing it at the message processing layer is an option is well, but that feels like a step in the wrong direction)
< cfields>
jonasschnelli: dummy handshake?
< jonasschnelli>
cfields: we do a normal unencrypted version/verack handshake, .. then p2p initiator sends encinit
< cfields>
i only bring it up because it seems avoidable, not because it's really a big deal
< jonasschnelli>
A... I guess now I understand your point...
< * jonasschnelli>
thinking
< cfields>
jonasschnelli: i'm not following. Net would still have to be told to switch to encryption.
< jonasschnelli>
Yes. It's another issue...
< jonasschnelli>
Wouldn't it be a problem for some clients if the first package would be a pubkey?
< jonasschnelli>
Also,.. how would you know if the other side supports encryption?
< cfields>
jonasschnelli: yea, I trivialized that part for the sake of the example :)
< jonasschnelli>
Heh... yes.
< jonasschnelli>
I guess leaving out the message processor is probably hard,...
< jonasschnelli>
eventually we need to look at the protocol version in the handshake and only send an encinit message if the protocol version signals support
< jonasschnelli>
Assuming that non-supported message types are just ignore may be a fragile concept
< jonasschnelli>
(though unsure,... its probably okay)
< cfields>
jonasschnelli: could just send some magic before the pubkey, which would signal support and require no parsing other than an equality check.
< jonasschnelli>
But if you connect to a peer and send magic+33b-pubkey+cipertype, woudln't you get disconnected straight away?
< jonasschnelli>
(if non BIP151 supporting peer)
< cfields>
uhmm.. IIRC we allow more than one chance for the initial message? checking.
< jonasschnelli>
Also,.. not sure if we should respect other implementations.
< jonasschnelli>
(libbitcoin / btcd / bcoin)
< luke-jr>
jonasschnelli: that's a disturbing thing to say.
< jonasschnelli>
luke-jr: I'm only saying since there are no clear protocol specification for this
< jonasschnelli>
(is it allow to send data before the version/verack handshake)
< luke-jr>
oh, I misinterpreted you I think
< jonasschnelli>
(then s/allowed/tolerated)
< jonasschnelli>
I mean ethically, we should respect them. :)
< cfields>
haha :)
< jonasschnelli>
But we where talking on possible encinit p2p encryption disconnets
< cfields>
jonasschnelli: ok, looks like we can send before version as long as the magic is correct.
< cfields>
but yes, that's very much an implementation detail. No clue how other clients handle it.
< jonasschnelli>
Yes. That sound a bit fragile...
< jonasschnelli>
But we could reconnect once we got disconnected after that pre-handshake encryption request and try without encryption (if unencrypted connections are allowed)
< cfields>
jonasschnelli: mm, it's not worth going that far I think
< Alexandra>
holaaaaa?
< gmaxwell>
jonasschnelli: we can coordinate if there is some incompatiblity.
< gmaxwell>
And we should avoid gratitiously breaking, where its possible without significant harm.
< gmaxwell>
But if at the end of the day an implementation which is almost non-existant on the public network needs updates to work right with sensible behavior, ... well then it is what it is. To be kind we could offer patches.
< gmaxwell>
I'm really not a fan of the "try then retry if it fails" behavior, being stateful makes it more complex and have more ways to fail.
< gmaxwell>
E.g. say the remote party is almost out of sockets, first try fails due to crypto snafu, second try just doesn't connect due to socket exhaustion.
< echeveria>
jonasschnelli: gmaxwell: cfields: personally I'd do something a little more interesting. bind a new socket, say 8383 and *only* accept encrypted connections there. older implementations explicitly avoid non-standard ports, and it gives the option to selectively only use bip151.
< gmaxwell>
echeveria: the downside there is that now everyone's firewall deployments need to change. :(
< echeveria>
there's already logic for multiple bind interfaces with different properties (ie, whitebind). all this means is you gossip both of your ports.
< gmaxwell>
one thing we could do in a slower migration is support two ports, one is legacy or encrypted, the other is encrypted only.
< gmaxwell>
but I'm still skeptical that it's worth doing that.
< cfields>
yea, I've considered that as well. But I was afraid that we'd get too much backlash from network admin
< cfields>
^^ what gmaxwell said
< gmaxwell>
one could, on the legacy port, hang up whenever encryption isn't used.
< cfields>
also, we currently penalize non-8333 I believe.
< gmaxwell>
that sounds silly now, but in two years when 99% of everything has BIP151 it'll be reasonable.
< echeveria>
cfields: that's sort of why it's ideal.
< echeveria>
cfields: bitcoin nodes won't ever reasonably connect to a non 8333 rumored port.
< gmaxwell>
the idea there being that old peers won't rumor or connect to the encrypted ones...
< cfields>
hmm. well I'd be all for it if we could get an idea of how many people it would piss off
< gmaxwell>
echeveria: so what really does that gain over just the disconnection approach? I think only that old peers will waste less time trying to connect to encrypted only nodes.
< cfields>
also makes me curious about the reasoning for the historical http/https 80/443 split. I actually have no idea how that unfolded initially.
< gmaxwell>
But I think there is relatively little reason to run encrypted-only for inbound. (for outbound connections, you don't need another port, service flags are sufficient)
< echeveria>
gmaxwell: I like moving away from 8333 that's shared with bcash, and being able to be selectively restrictive about the type of traffic without necessarily inspecting it is nice. to me it's just easier to reason about, but that's possibly personal preference.
< gmaxwell>
effectively, I think echeveria's proposal is to abuse the port number to create an implicit "old nodes don't connect here please" flag.
< echeveria>
yup.
< cfields>
yea. which I guess is exactly the net-level signal that I'm after.
< gmaxwell>
Probably that idea should get incorporated in wumpus' new addr message proposal. e.g. define some set of service flags where if you don't know their meaning you avoid connecting.
< gmaxwell>
doesn't help old peers, but we'll probably have this desire in the future.
< cfields>
gmaxwell: lol, we could use 18333 :)
< gmaxwell>
Also, I'd rather bother network admins just once when we add a UDP based transport in the future. :)
< cfields>
(not a serious suggestion, but it would make life a bit easier on the routing side)
< gmaxwell>
I dunno if pieter has talked to y'all about what we came up with for authentication, it's pretty awesome. We gain the property that an active MITM cannot detect if authentication is actually in use or not, so he can't monitor anyone at all or takes the risk that his interception is detected. This is a major advance over auth everywhere else where an active MITM can detect users using auth and
< gmaxwell>
just sever the link (like an innocent network failure) and stop MITMing between that user pair.
< sipa>
i have
< sipa>
though the best we have is something that only supports one pubkey and one privkey afaik
< cfields>
gmaxwell: yes, that's very cool.
< cfields>
sipa: thoughts on echeveria's suggestion to use a separate port?
< sipa>
cfields: seems a nice and easy solution, but i see the possible administrative issues
< sipa>
i also don't think there is that much of a problem with reconnecting
< gmaxwell>
I think the only thing a new port would buy is avoiding old peers attempting to connect to us if we were only going to accept encrypted connections on inbound. Or am I missing some other benefit?
< sipa>
well there are 3 approaches suggested (a) new port (b) reconnect on same port with wholly different protocol after learning peer supports encryption (c) upgrade the connection in flight
< cfields>
gmaxwell: that's it, but it makes traffic significantly easier to handle imo
< gmaxwell>
cfields: how so?
< cfields>
sipa: (bi) always plan to reconnect to upgrade (bii) only do it if they boot us for trying
< gmaxwell>
I think I'm confused. AFAIK the 'boot us for trying' isn't a real problem.
< cfields>
gmaxwell: as described, bip151 would require us to send the first bytes up for parsing, then have the message handler tell the net handler to deal with encryption from that point on. If encryption could be assumed, net could just handle it transparently.
< gmaxwell>
This is perhaps a sign that our layering is faulty. :P
< cfields>
and this is me trying to avoid falling in the same trap again :)
< gmaxwell>
but okay, I see that. I think forcing onto another port due to software layering in the networking is more than a little ugly.
< cfields>
er... I say "net layer" and "message processing layer", but I mean those conceptually. Not our specific implementation.
< gmaxwell>
if encryption must be on another port, rather than optionally on another port then we'd get a massive deployment loss due to users needing to punch new firewall holes.
< gmaxwell>
E.g. I was thinking of echeveria's proposal as 8333 becomes legacy OR encrypted, and 8334 is encrypted only.
< cfields>
Ah, ok.
< cfields>
well, I suppose we could do that as well. Introduce the round-trip layer violation for 8333, and hope for some future that's nearly all 8334, and un-encrypted 8333 could be dropped at that point.
< cfields>
I guess that doesn't provide much motivation to switch, though.