< bitcoin-git>
[bitcoin] gkrizek opened pull request #15255: [tests] Remove travis_wait from lint script (master...remove-travis_wait) https://github.com/bitcoin/bitcoin/pull/15255
< jonasschnelli>
Is there another reason then complexity that the script descriptors do not have a range specifier? Like wpkh(xpub6DJ2dNUysrn5Vt36jH2KLBT2i1auw1tTSSomg8PhqNiUtx8QX2SvC9nrHu81fT41fvDUnhMjEzQgXnQjKEu3oaqMSzhSrHMxyyoEAmUHQbY/0/*[100-200])?
< sipa>
jonasschnelli: yes, not all places where descriptors are useful require a range
< sipa>
specifically, a wallet doesn't, as it can automatically extend the ranges
< jonasschnelli>
sipa: but it could be an optional element that may be ignored by the application logic
< jonasschnelli>
sipa: say I want to import p2wpkh script from an xpub from 100 to 200
< sipa>
then you need a record with that descriptor, and that range
< sipa>
there is other metadata you need that has no place in a desceiptor
< kallewoof>
looks like adding 'osx' as an os is possible in travis (but would result in 2* the number of builds, if I understand... and there is a bunch of docker stuff in there now that wasn't there when I fiddled last..)
< sipa>
kallewoof: ideally we'd be able to test the linux cross-compiled macos binaries... but that's tricky as it means tranferrings builds from one run to another
< jonasschnelli>
Maybe as a starter just run one build (wallet + QT) on nativ travis osx and run the tests?
< jonasschnelli>
Running the cross compiled macos binaries would indeed be great...
< jonasschnelli>
But since macOS is very likely non apple nativ and even if – its a form of a BSD, we could eventually use the same cross compile toolchain used under linux to somehow emulate the cross compile process and run the test
< sipa>
what does "since macos is very likely non apple native" mean?
< jonasschnelli>
the travis macOS is probably an emulated macOS... not a Apple native... probably some sort of Hackintosh
< jonasschnelli>
the test results are eventually not directly identical... but I may be wrong.
< jonasschnelli>
AFAIK Apple (by the TOC) does not allow virtualization of its macOS on non mac hardware ... and I don't expect that travis has a bunch of iMac laying around
< jonasschnelli>
kallewoof: Yeah. I have read this... but I wonder how they do it
< jonasschnelli>
macOS runs only on mac
< jonasschnelli>
expect you modify it in a pretty nasty way
< jonasschnelli>
(which I what makes me say the test results may not be completely representative)
< sipa>
they may well have negotiated a license agreement with apple
< jonasschnelli>
could be... but unlikely
< jonasschnelli>
But they maybe have a couple of mac pros or so running those jobs
< echeveria>
jonasschnelli: it’s not modified much actually.
< jonasschnelli>
AFAIK there are no drivers for non mac hardware, etc.
< jonasschnelli>
echeveria: what means "not much"? :)
< echeveria>
jonasschnelli: there’s a kernel module, DONTSTEALMACOS.kext, and other than that it’s just driver support mostly.
< echeveria>
most services just virtualize OS X on a farm of Mac Minis anyway, which doesn’t violate the license and sort of fits into a data center.
< jonasschnelli>
Yeah. I heard about this. But I guess not very efficient...
< jonasschnelli>
Mac Mini is probably the nightmare of a server data center admin...
< sipa>
it looks like they run on actual mac hardware
< echeveria>
they run mostly alright for this sort of target. I know a developer that has a dedicated Mac Mini build server for their binary distribution in a similar way to bitcoin core.
< echeveria>
similar to Travis, building Bitcoin Core, I men’s.
< jonasschnelli>
Good to know...
< echeveria>
mean.
< jonasschnelli>
however, a travis macOS job would probably be a good thing...
< sipa>
agree.
< jonasschnelli>
Curious to know if you can run the GUI process and take a screenshot...
< bitcoin-git>
[bitcoin] Empact opened pull request #15258: Scripts and tools: Fix devtools/copyright_header.py to always honor exclusions (master...copyright-header-abs) https://github.com/bitcoin/bitcoin/pull/15258
< kallewoof>
I can't make the meetings as always, but would like to add #13756 to priority for review list, unless people object. (ping wumpus fanquake ...)
< bitcoin-git>
[bitcoin] domob1812 opened pull request #15259: Use real HTTP bind address in curl RPC help (master...decoupled-rpchelp) https://github.com/bitcoin/bitcoin/pull/15259
< bitcoin-git>
bitcoin/master f618c58 Ben Woosley: Docs: Update python docs to reflect that wildcard imports are disallowed
< bitcoin-git>
bitcoin/master ab46fe6 MarcoFalke: Merge #15249: Docs: Update python docs to reflect that wildcard imports are disallowed...
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #15249: Docs: Update python docs to reflect that wildcard imports are disallowed (master...wildcard-imports-doc) https://github.com/bitcoin/bitcoin/pull/15249
< bitcoin-git>
[bitcoin] sipa opened pull request #15263: Descriptor expansions only need pubkey entries for PKH/WPKH (master...201901_flatprovider_pkh) https://github.com/bitcoin/bitcoin/pull/15263
< provoostenator>
Do we have a wallet meeting today or am I off by one again?
< achow101>
wallet meeting?
< sipa>
present
< achow101>
<meshcollider> I'm not sure whether I'll have internet at the start of the wallet meeting, so sipa can you host it this week please :)
< provoostenator>
Suggested topic: wallet goals for 0.18
< gmaxwell>
00:45:35 < kallewoof> I can't make the meetings as always, but would like to add #13756 to priority for review list, unless people object. (ping
< sipa>
gwillen: will you have time in the near future to rebase/address comments in #14978 ?
< provoostenator>
I'd also love to get a number of pull requests into this release that would allow using achow101's HWI library:
< gribble>
https://github.com/bitcoin/bitcoin/issues/14978 | Factor out PSBT utilities from RPCs for use in GUI code; related refactoring. by gwillen · Pull Request #14978 · bitcoin/bitcoin · GitHub
< provoostenator>
Though that may be a bit ambitious.
< achow101>
how likely is it for #14491, #14075, and #14021 to get into 0.18?
< gribble>
https://github.com/bitcoin/bitcoin/issues/14075 | Import watch only pubkeys to the keypool if private keys are disabled by achow101 · Pull Request #14075 · bitcoin/bitcoin · GitHub
< provoostenator>
I commented "I would prefer if TopUpKeyPool can deal with imported keys.", though I'm not sure how realistic that is without a complete wallet descriptor support redo.
< gmaxwell>
Is there anything we can do to head off funds loss from typos/copypaste errors with decriptor importing? The use of descriptors as user/api handled key material seems like a step backward for Bitcoin, as we're now introducing ways for simple typos (or clbuttic errors) to cause funds to go off to space.
< provoostenator>
gmaxwell: we could make importing descriptors without xpub fail, unless opted in?
< gwillen>
achow101: sweet, thanks! I will go look.
< provoostenator>
(xpub or other checksummed approach)
< gwillen>
sipa: ok, I will prioritize, thanks
< sipa>
gmaxwell: the xpubs inside descriptors have a checksum, though the paths/functions don't
< gmaxwell>
provoostenator: if the part of the descriptor outside of the xpub is corrupted funds are still lost.
< gmaxwell>
That may be a little less likely, but its still exposed.
< provoostenator>
That seems somewhat unlikely though for the most basic descriptors.
< provoostenator>
Because the parser will fail, unless the error happens to produce another valid descriptor.
< gmaxwell>
What does wsh(multi(3,xpub661MyMwAqRbcFW31YEwpkMuc5THy2PSt5bDMsktWQcFF8syAmRUapSCGu8ED9W6oDMSgv6Zz8idoc4a6mr8BDzTJY47LJhkJ8UB7WEGuduB/1/0/*,xpub69H7F5d8KSRgmmdJg2KhpAK8SR3DjMwAdkxj3ZuxV27CprR9LgpeyGmXUbC6wb7ERfvrnKZjXoUmmDznezpbZb7ap6r1D3tgFxHmwMkQTPH/0/0/*)) do?
< gmaxwell>
(3 of 2 multisig)
< provoostenator>
We could use the Electrum style xpub, ypub, zpub, etc-pub to indicate what type we expect the descriptor to be. But there's no software that could produce the data.
< provoostenator>
(a generic problem if we had any kind of checksum mechanism to the descriptor language)
< gmaxwell>
in any case, filtering to xpubs unless overridden seems like a good idea, at least.
< gwillen>
given that the xpubs themselves already have a checksum, you could add a checksum of _just_ everything else to the end, it could be very short and still be sufficient
< sipa>
also, anything with a private key has a checksum
< gmaxwell>
but it still seems far too dangerous to ever have users using directly to import keys.
< provoostenator>
3 of 2 multisig should hopefully fail as an invalid descriptor? Could be worth adding a test for that...
< provoostenator>
gwillen: the problem is, what software will make the checksum? We'd have to define a standard for that too. Maybe that's a good thing.
< gmaxwell>
one could adjust the descriptor language to support a ",checksum" at the end. Even if it just blindly supported them with the checkvalue truncated to facilitate editing, it would provide some protection against other corruption. I dunno.
< gwillen>
provoostenator: yeah, we would have to go "oops, lack of checksum was an omission in the spec, new spec"
< gmaxwell>
I don't really understand how it got to this point. When descriptors were first defined, I didn't expect them to be used for key management in liue of importing addresses.
< gwillen>
which makes sense if they are going to be an interchange format, which it seems they are otherwise well-suited for, except this issue
< sipa>
gwillen: that's the great thing about not having a spec
< achow101>
provoostenator: it's not like descriptors are really standardized yet though...
< gwillen>
sipa: :D
< gmaxwell>
I do worry about breaking editability. Though there could just be a simple utility(function) to recompute the checksum on any descriptor.
< gmaxwell>
That you'd just use after editing, if you intended to edit.
< provoostenator>
gmaxwell: the optional ,checksum makes sense. And then importmulti could refuse descriptors without such checksum, unless "I-know-what-I'm-doing" is added to the command.
< sipa>
provoostenator, gmaxwell: that seems reasonable
< sipa>
i don't mind breaking compatibility at this point
< provoostenator>
Yeah, we would need some utility to calculate the checksum for those making descriptors manually.
< gmaxwell>
Seems straightforward.
< sipa>
gmaxwell: i think i imagined initially that indeed there would be syntactic sugar encodings for common subsets of descriptors (like zpub etc) with checksums etc
< sipa>
but it seems people (myself included) are pretty excited about having the descriptor itself be a visible thing... so adding a checksum to the descriptor itself makes sense
< gmaxwell>
Better to fix it now, if not later. I wonder if we need the 'accept import override' if there is a utility function that just adds the checksum?
< gmaxwell>
Yea, I think they're cool, I'm excited about having them available too. Just dreading the footgun. But it seems there is a solution.
< gwillen>
the voice in the back of my head worries about the checksum being an optional thing stuck on the end, that people will strip it or ignore it
< gwillen>
but I don't have a better idea in hand
< * sipa>
fires up BCH code search?
< gwillen>
there's no obvious better place to put it
< gmaxwell>
gwillen: well we could mandate it, and not have anything that ignores it, but have a utility function that regenerates.
< gmaxwell>
If someone wants to do extra work to screw themselves, oh well.
< instagibbs>
Put bytes in pseudorandom places, annoying to extract and by then, might as well validate it
< instagibbs>
(sarcasm font)
< gmaxwell>
But the default should be as safe as we can make it without undue tradeoffs.
< gmaxwell>
sipa: whats the character set of descriptors?
< provoostenator>
Or you could design the descriptor language such that there's no way to accidentally break it with a N character mistake.
< gmaxwell>
sipa: (does it form a field...)
< gmaxwell>
provoostenator: it's easy to do that if the number of candidate characters is a prime power. Otherwise we only know how to make it immune to a 1 character mistake.
< provoostenator>
I'm in favor of prime power.
< gmaxwell>
I don't think right now descriptors have a defined charset?
< sipa>
gmaxwell: indeed
< sipa>
but at least 0-9 a-z A-Z / * [ ]
< gmaxwell>
alphanum + some extras like /,()[]* ?
< gmaxwell>
oh right mixed case.
< provoostenator>
Pubkeys and privkeys have defined charset, so you can include the checksum part of those in the total checksum? (or just checksum the whole thing)
< gwillen>
can't you always just add "virtual" characters to the charset until you reach a prime power
< gwillen>
at the expense of increasing checksum size a bit
< gmaxwell>
Checksum can be defined over a smaller charset, with a folding routine so that outside of character set characters get treated as some inside set character for checksum purposes.
< gmaxwell>
gwillen: no, because the check digit could be one of those characters.
< gmaxwell>
You could however make the checksum base 25, encoded to alpha, and then fold all other characters to it... in any case, probably not something to hash out in the meeting.
< gmaxwell>
and doing something dumb (sha256, ugh) would be still better than nothing.
< sipa>
we can reuse bech32 character set easily
< sipa>
and use something like what bech32 does for the prefix
< gmaxwell>
sipa: with the hrp han... right...
< sipa>
(which works for all of ascii)
< gmaxwell>
doesn't have the great distance properties, but probably good enough.
< sipa>
before we move to a different topic... what general length of checksum are people thinking is acceptable?
< gwillen>
how would people feel about the syntax being something like "(descriptor,checksum)" instead of "descriptor,checksum"
< gwillen>
adding two extra characters is kind of silly but it feels to me like it would make the checksum less likely to get lost in copy-paste confusion
< gmaxwell>
sipa: descriptors are already long, I don't see a problem with adding 5-8 extra characters.
< gwillen>
(also, the checksum charset should exclude "()[]*/.", it should be alphanum, for the same reason)
< provoostenator>
Sticking to bech32 characters would be nice.
< provoostenator>
Because longer term it's probably nice if desciptors can be printed as QR codes.
< provoostenator>
I'm thinking paper backups.
< provoostenator>
(of pub keys)
< gmaxwell>
I guess spaces are syntatically meaningless in descriptors? if so checksum should be computed after stripping them.
< instagibbs>
spaces are currently rejected anywhere
< instagibbs>
in Core
< gwillen>
oof, it would be good if descriptor format were canonical, no spaces
< gmaxwell>
provoostenator: well unfortunately the other characters in descriptors make QR codes inefficient.
< gwillen>
:+1:
< gmaxwell>
oh okay, cool.
< sipa>
gmaxwell: i think the parser will reject anything with spaces
< instagibbs>
sipa, it will(I've lost time figuring this out)
< instagibbs>
the error message is quite vague
< gmaxwell>
instagibbs: improve error messages?
< instagibbs>
gmaxwell, yeah
< gmaxwell>
sorry for the derail, but I'm glad there seems to be support for fixing this.
< sipa>
gmaxwell: fwiw, your 3-of-2 multisig descriptor is rejected
< instagibbs>
I mean you might accidentally do a wrong number or something, 1-of-2 instead of 2-of-2, similar failure
< gmaxwell>
sipa: thats great.
< gmaxwell>
or 2 of 2 instead of 1 of 2.
< gmaxwell>
sipa: is 0 of 2 rejected? :P
< sipa>
and i think p2sh descriptors with multisig redeemscripts over 510 bytes are also rejected
< sipa>
gmaxwell: yes, 0 of 2 is also rejected
< gmaxwell>
The fact that a lot of mistakes will fail the parser also favors a weaker check.
< sipa>
ok, let's move to a different topic
< sipa>
if there are others?
< gmaxwell>
Is anyone working on improving avoidpartialspends?
< jnewbery>
for wallet goals for v0.18, I'm hopeful we can finish multiwallet in the GUI
< jnewbery>
promag's been doing great work
< gmaxwell>
It's off by default, which makes it kinda useless in terms of overall privacy effects.
< sipa>
gmaxwell: what needs to be done?
< gmaxwell>
sipa: It needs to grow a value threshold. "Avoidpartialspends unless the fee would increase by more than X" which we could ship enabled.
< gmaxwell>
And privacy maniacs could turn it on unconditionally, but everyone would at least be happy with a threshold of 0. :P
< gmaxwell>
(I think we should set the threshold by default to the dust limit or something similar, the exact value doesn't matter too much)
< gmaxwell>
Reminder: this is the wallet feature that causes it to try to spend all payments to a reused address at once.
< gmaxwell>
Doing so can cause higher fees, so its off by default, and as a reasult I doubt anyone uses it.
< gmaxwell>
result*
< sipa>
right
< sipa>
i agree that makes it pretty pointless
< gmaxwell>
sometimes it doesn't even increase fees at all, I know of now reason why people wouldn't prefer it in at least that case. But even when it causes higher fees, usually it's just pulling in fees from the future, not really paying more.
< gmaxwell>
This returns to a larger open question about fees now vs fees later.
< sdaftuar>
gmaxwell: one concern i had with unconditional use of avoidpartialspends was edge cases with small value inputs. eg i dust your wallet with junk, and cause you trouble
< sdaftuar>
i think we mitigated that mostly, maybe entirely, with a cap on the number of inputs we could pull in?
< gmaxwell>
and/or with a threshold you'll pay in extra fees?
< sdaftuar>
yeah, we should just do that.
< gmaxwell>
but I agree there should be a cap on the group size, but I'm not sure how best to do that.
< gmaxwell>
For the longer term question, I think eventually we want the wallet fee estimation to have an idea of "fees expected to be higher/lower in the future", and in response we should favor or disfavor including more inputs.
< sipa>
gmaxwell: don'
< sipa>
gmaxwell: don't we already have that?
< sipa>
(using long term fee estimation)
< gmaxwell>
I think we do in one really narrow case (we use the long term fee for estimating the cost of spending the change output in BNB)? I don't think we have it more generally.
< gmaxwell>
I think because fees have been low most of the time in the last 6 months not a lot of attention is going to them, ... which is unfortunate at least in terms of right now would be a great time for wallets to be agressively aggregating.
< gmaxwell>
(well not litterally right now, there are fees at the moment.. :P)
< sipa>
right
< gmaxwell>
so okay well that would be a bonus feature for avoidpartial: use long term vs current to switch between threshold vs always on.
< gmaxwell>
If fees are high only be willing to pay slightly more, if fees are low use it.
< jnewbery>
BitGo claim to already do 'predictive UTXO management' (ie use more inputs when fees are expected to be higher in future, and fewer inputs when fees are expected to be lower in future)
< gmaxwell>
it's been discussed a lot in the past, it's just not clear how well it can work without a human hand available to rescue it if it becomes stupid.
< gmaxwell>
It's a lot easier to do smart stuff if you can count on an expert non-artifical-intelligence to override if it goes dumb, you don't have to solve ever corner case or potential avenue for abuse.
< bitcoin-git>
[bitcoin] jamesob opened pull request #15264: validation: remove useless uncache accounting in ATMPW (master...2019-01-remove-useless-uncaching-atmpw) https://github.com/bitcoin/bitcoin/pull/15264
< sipa>
last half minute topic?
< sipa>
#endmeeting
< lightningbot>
Meeting ended Fri Jan 25 20:00:25 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< bitcoin-git>
[bitcoin] sdaftuar opened pull request #15265: Flush without erasing cache during periodic and pruning flushes (master...2019-01-periodic-flush-dont-wipe) https://github.com/bitcoin/bitcoin/pull/15265
< gmaxwell>
talk of multiwallet gui makes me wonder if anyone is working on using BIP157 filters for rescan? Personally I found multiwallet not super useful, due to the need to rescan wallets that were left unloaded, and it taking 8 hours to do so...
< sipa>
gmaxwell: jimpo was working on that
< provoostenator>
gmaxwell sipa: re descriptor checksums, maybe we should just keep human readable descriptors without a checksum, and define a separate (bech32) serialization that has the same information.
< provoostenator>
And then recommend that computer programs don't pass around the human readable versions.
< sipa>
provoostenator: i was considering that too
< provoostenator>
(or we can do both, i.e. also adding a checksum for the human readable one)
< gmaxwell>
that seems unlikely to work, unless you'd actually suggest we refuse to accept the human readable ones as input?
< provoostenator>
But that also solves my checksum concern.
< sipa>
but i think it won't be used really, as it removes the human readability
< sipa>
and i think that reafability is part of the appeal
< provoostenator>
I think it's still useful for communication between programs.
< sipa>
another question is what length the code should be designed for
< provoostenator>
"But that also solves" (not sure what I as trying to say)
< gmaxwell>
sipa: "long" or at least a code which is distance 1 for arbritarily long would be good.
< sipa>
1024 won't be long enough as a maximum
< sipa>
gmaxwell: every code is distance 1 ;)
< sipa>
(including no checksum at all)
< provoostenator>
I can't actually think of that many situations where I _want_ to type descriptors.
< sipa>
provoostenator: agree, but i may want to compose them
< sipa>
and copy paste them
< sipa>
gmaxwell: and every bch code is distance 2
< sipa>
at infinite length
< sipa>
the question is up to what length it is distance 3 and higher
< provoostenator>
Yes, composing makes sense. That's where plain text helps, and checksum is by definition pointless, because your brain is the source.
< provoostenator>
And calls like getaddressinfo should probably show both the human readable and the bch version
< jim_layhee>
hello all, I'm trying to run bitcoind in regtest. configured using ./configure --without-gui --disable-wallet on osx and i get the following error when trying to start using ./bitcoind -regtest -daemon
< jim_layhee>
Error: Unable to start HTTP server. See debug log for details.
< jim_layhee>
any ideas?
< promag>
thanks jnewbery
< promag>
sorry for the lack of updates.. had a bad week
< sipa>
(note that one character error may cause 2 weight difference due to expansion)
< gmaxwell>
sipa: re earlier, I thought to support detecting one error at arbritary lengths the code had to have a +1 term.
< sipa>
gmaxwell: no
< sipa>
you're thinking off the ability to detect an arbitrarily large odd number of errors
< gmaxwell>
derp ah
< gmaxwell>
The fact that wanting a smaller checksum charset than the rest (to avoid including delimiters) results in error expansion is kind of annoying.