< gmaxwell>
https://blog.bitmex.com/bitcoins-consensus-forks/ has anyone already pinged these folks and pointed out that stuff like the addition of NOPs was not a hardfork? (prior to that commit all unknown ops were nops)
< gmaxwell>
re that page, really anything that wants to say that the system has hardforked previously ought to be explaining why the very first release can validatate past their claimed hardfork.
< gmaxwell>
because the traditional definition of a hardfork (as used on that article) doesn't really allow for that.
< luke-jr>
gmaxwell: most consensus rules are never triggered. that's immaterial to the change.
< gmaxwell>
it's pretty material to the question of it being debatable if there was a hardfork or not!
< luke-jr>
if the rules were relaxed, it was by definition a hardfork.
< gmaxwell>
I mean we don't actually know if the script split was a hardfork because at most its a latent hardfork, no one has constructed a demonstration case.
< luke-jr>
no demonstration case is needed
< gmaxwell>
Sometimes it's really hard to tell if they were relaxed or not.
< luke-jr>
which one are you referring to?
< gmaxwell>
" I mean we don't actually know if the script split was a hardfork"
< luke-jr>
oh, 0.3.7? I guess that's unclear.
< gmaxwell>
for years we thought it wasn't then someone came up with a credible argument that it might have been.
< gmaxwell>
But it's complicated enough that it probably won't be convincing without an example transaction.
< luke-jr>
couldn't an OP_IF span the two scripts previously?
< luke-jr>
hmm, can't think of a way to make that do invalid->valid, nm
< gmaxwell>
I think the example required an invalid serialization whos bytes gobbled up the code seperator.
< gmaxwell>
But AFAIK no one has ever tested it, and that might not work in the old code for some not immediately obvious reason.
< gmaxwell>
e.g. you end the scriptsig with a multibyte push opcode and an invalid length on the script so when concatinated with the code sep the codesep becomes part of the push.
< gmaxwell>
or something along those lines.
< luke-jr>
right
< luke-jr>
not sure it's worth the effort to prove either way, maybe it can just say "unclear"
< gmaxwell>
in any case, when people talk about hardforks, they ususally believe it disrupted the network and split consensus, but a latent hardfork hasn't done that.
< gmaxwell>
Which is why I think it would be useful to signal when they're latent or not and when (if ever) they materalized.
< luke-jr>
only people who don't understand hardforks, since a real one wouldn't disrupt the network nor split consensus
< gmaxwell>
luke-jr: yea, so only to 99.99999% of people on earth. :P
< luke-jr>
most people don't have either a correct understanding NOR a misunderstanding ;)
< gmaxwell>
So for example for the 2013 thing I'd say that it was a latent hardfork that became sensible in sept 2014 (or whenever old versions were actually reliably forked off).
< luke-jr>
I'd say all hardforks are "latent"
< luke-jr>
if the network splits, it would cease to be a hardfork, and become an altcoin like BCH instead
< gmaxwell>
Even if the network hasn't split, the more permissiveness could have been used or not.
< gmaxwell>
Imagine this, say the network rules change so that any coin with pubkey P could also be spent with pubkey P' = hash_to_point(P). This is a hardfork, but unless ECC is broken it can never be anything but a latent hardfork. It would be a dumb change for sure, but it wouldn't imply basically any of the risks of a hardfork.
< gmaxwell>
in any case, for the list the NOP thing was just a softfork (AFAIK), I think a few other early softforks are missing.
< luke-jr>
you want to fix it, or should I? ;)
< gmaxwell>
The script split thing I'd describe as a "possible latent hardfork [1]" "[1] The change wasn't believed to be a hardfork for years, but later some people conjectured some contrived transaction style which might be valid under the new rules but not the old, but it's complicated enough no one has bothered to check for sure, especially since any impacted software was long since no longer in use."
< luke-jr>
actually, I'm not sure I see how the invalid script thing could be used to make something valid that was previously invalid.. seems the end result would be invalid after a split too
< luke-jr>
because then you have a chopped opcode to execute
< gmaxwell>
for the 2013 thing, it's a "hardfork kinda" because the old behavior was non-determinstic. If you use your definition of a hardfork where a latent hardfork is a hardfork, then the 'time' of that hardfork was the initial release; since any given node could permit things others would forbid.
< luke-jr>
afaik, the new rules after 2013 May permitted things that would never have been permitted previously, at least by any naturally functioning node
< gmaxwell>
luke-jr: I dunno I'd have to find the discussion. I recall thinking that it was plausable.
< luke-jr>
maybe kanzure remembers XD
< gmaxwell>
luke-jr: I'm not _totally_ sure of that. I've synced as far as sept 2014 but the place where it fails was different between different runs. Maybe if I tried millions of times it would make it up to current.
< luke-jr>
gmaxwell: even if it did, you could still construct a block that is valid under the new rules that it would reject
< gmaxwell>
in any case, a change in behavior from highly non-deterministic to slightly more permissive determnistic is not your 'typical' hardfork. :P e.g. "do nothing" wasn't an option.
< gmaxwell>
luke-jr: thats what I'm not sure of. I don't know if you actually can construct a block that the old code will _never_ accept, only that it's very unlikely to accept.
< luke-jr>
the fix to the problem, in March, was to make the rules deterministically stricter ;)
< luke-jr>
ie, semver 1.0.2 on the wiki page
< gmaxwell>
luke-jr: I'm pretty confident that the temporary softfork wasn't actually enough to protect nodes from the non-determinism, FWIW.
< luke-jr>
why not? wasn't it strictly more restrictive?
< gmaxwell>
No, because the BDB behavior was more complicated than we understood at the time.
< luke-jr>
>_<
< gmaxwell>
I think, in fact, depending on what read activity was going on, it could have run out of locks for a block that only touched two txids (e.g. a single transaction).
< gmaxwell>
at least in theory.
< luke-jr>
you mean in the wallet bdb?
< luke-jr>
or what else would cause reads going on?
< gmaxwell>
indeed.
< luke-jr>
well, the limits had to be strict enough that it could survive at least a minimal reorg, so I'd be surprised if the wallet could interfere too much in normal operation
< sipa>
gmaxwell: addition of NOPs was at the same time as removal of VER
< gmaxwell>
sipa: yes, and? removing ver was a softfork.
< gmaxwell>
(I suppose luke should argue that every version _prior_ to that point was a hardfork, due to OP_VER. ...)
< gmaxwell>
luke-jr: Another reason why latent vs non-latent hardforks matter: a latent one could still be softforked out.
< sipa>
gmaxwell: script split was a trivial hardfork; the sighash effect of codesep changed
< gmaxwell>
e.g. say you did come up with a transaction pattern that split off 0.3.0, but yet was never used in the network so far. That pattern could be softforked out and then the 'hardfork' is undone.
< luke-jr>
gmaxwell: any hardfork can be softforked out.
< gmaxwell>
luke-jr: not in a way that prevents older software from going out of consensus.
< luke-jr>
?
< sipa>
gmaxwell: did you see my comment on sighash and codesep?
< gmaxwell>
sipa: yes, so you're saying that any transaction with a codesep in the scriptpubkey that is valid now wouldn't have been valid pre-0.3.0 ?
< gmaxwell>
(well codesep and a valid signature, of course)
< sipa>
gmaxwell: yes
< sipa>
a signature with a checksig in the scriotsig
< gmaxwell>
now I'm wondering if the point I was unable to get 0.1.0 ultimately past was really a codesep use and not related to the locks thing at all.
< sipa>
it may be impossible to construct a valid one like that now...
< sipa>
maybe when using sighash_single bug
< gmaxwell>
it would certantly make more sense, since the block in question wasn't even especially large.
< gmaxwell>
luke-jr: in any case there is also a big difference between intentionally adding a hardfork and accidentally adding one in a crazy corner case when trying to fix a bug that let anyone spend any coin. :P
< luke-jr>
gmaxwell: depends on why you're classifying
< sipa>
Dhiraj: either open an issue, or email security@bitcoincore.org
< Dhiraj>
okay cool, thank you
< harding>
sipa: the difference is which side of a two-way merge it follows. ^ is the left side (main code side by default), ~ is the right side (merged-in side) IIRC.
< Dhiraj>
Thanks mail sent too security@bitcoincore.org
< sipa>
harding: thanks
< sipa>
harding: just looked up it, not really
< sipa>
they're the same
< sipa>
^i means the ith parent
< sipa>
~i means the i times 1st parent
< sipa>
so HEAD~ and HEAD^ are the same; both refer to the first parent of HEAD
< sipa>
but HEAD~2 is the first parent of the first parent
< sipa>
while HeAD^2 is the second parent
< harding>
sipa: Hmm. (Looks it up myself.) Ok, yeah. This quote seems like a good way to remember it: "way to remember: '~' is "fuzzy" or "approximate" (i.e., you only get the first parent) while '^' is precise (so goes through every single commit)."
< bitcoin-git>
[bitcoin] AkioNak opened pull request #13711: [bench] Add benchmark for unserialize prevector (master...add_bench_unserialize) https://github.com/bitcoin/bitcoin/pull/13711
< bitcoin-git>
[bitcoin] practicalswift opened pull request #13712: wallet: Add error handling. Check return value of ParseUInt32(...) in… (master...check-ParseUInt32-return-value) https://github.com/bitcoin/bitcoin/pull/13712
< provoostenator>
I just had a node (on crappy hardware) complain "ERROR: ConnectBlock: CheckQueue failed" and "InvalidChainFound: invalid block=" on a block that's valid. It then just sat there complaining that its peers were giving it invalid tips.
< bitcoin-git>
[bitcoin] jonasschnelli opened pull request #13713: Ignore new blocks when -stopatheight target has been reached (master...2018/07/stopatheight) https://github.com/bitcoin/bitcoin/pull/13713
< provoostenator>
reconsiderblock did the trick and now it's happy again
< provoostenator>
For about 30 blocks, rinse and repeat. This probably falls under "there's no generic way to deal with crappy hardware". Maybe just recommend a virtual RAID for indexes and chainstate dirs :-)
< provoostenator>
Maybe worth noting that this only started happening after the assumevalid block and the CPU's are only 50C. I think a while ago we discussed the idea of retrying signature validation a few times to handle "cosmic rays".
< kallewoof>
Sure the hardware isn't broken?
< provoostenator>
I'm sure the hardware _is_ broken.
< kallewoof>
ok
< provoostenator>
Assume valid block is 506067, this starts happening at 506364. I'm just wondering what's reasonable in trying to compensate for bad hardware.
< bitcoin-git>
[bitcoin] ken2812221 opened pull request #13714: [gitian-build] Add automatical lxc network setup for Bionic (master...gitian-build-auto-install) https://github.com/bitcoin/bitcoin/pull/13714
< jonasschnelli>
wumpus: what do you think about #9662. IMO it's ready for merge (has >5 utACKs). Since it's my PR, I'd prefer if someone else merges it
< gribble>
https://github.com/bitcoin/bitcoin/issues/9662 | Add createwallet "disableprivatekeys" option: a sane mode for watchonly-wallets by jonasschnelli · Pull Request #9662 · bitcoin/bitcoin · GitHub
< bitcoin-git>
[bitcoin] ken2812221 closed pull request #13660: [depends] Don't build Qt dependencies if it doesn't support Qt (master...qt-packages-fix) https://github.com/bitcoin/bitcoin/pull/13660
< ttlloo>
s
< ttlloo>
getblonkNumber
< bitcoin-git>
[bitcoin] marcoagner opened pull request #13715: tests: fixes mininode's P2PConnection sending messages on closing transport (master...fix_mininode_p2pconnection_send_message) https://github.com/bitcoin/bitcoin/pull/13715
< arubi>
sipa, wrt earlier script{sig,pubkey} fork, do you mean that a script like " [script lhs] codesep <pubkey> checksig | [script rhs] " could be signed in a way that's invalid pre fork and valid after? here "script lhs" and "script rhs" are just any script and '|' is the point where scriptsig and scriptpubkey are concatenated, so a second codesep pre-fork.
< arubi>
at first I thought it was about codesep being "missing" to the right side of the sig post-fork, but afaict the signature would not sign the second codesep op pre-fork anyway because findanddelete removes it, so that's probably not it.. also sighash_single effectively just signs "1" so I don't see how anything in script might change what a sig must sign.
< arubi>
anyway, if you could expand on what such an invalid pre-fork and valid after script would look like, I'd be very interested :)
< arubi>
s/sighash_single/sighash_single bug/
< arubi>
I do like the malformed serialization thing.. it feels like you could massage the bytes pushdata takes to glob the codesep pre-fork.. that's really interesting
< bitcoin-git>
bitcoin/master a4ba238 Lawrence Nahum: depends: disable Werror for zmqlib release, causes ndk build to break
< bitcoin-git>
bitcoin/master 8c36432 MarcoFalke: Merge #13689: depends: disable Werror when building zmq...
< sipa>
arubi: yeah, i think you can exploit sighash_single on an input position above the number of outputs, which signs the message 0x0000...0001
< gmaxwell>
I didn't understand your sighash single claim myself.
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #13689: depends: disable Werror when building zmq (master...zmq_clang_depends) https://github.com/bitcoin/bitcoin/pull/13689
< gmaxwell>
since it's signing a constant, why would the data going into the sighash matter?
< sipa>
gmaxwell: uh, yes
< sipa>
of course :)
< arubi>
phew :)
< arubi>
so that leaves something like pushdata behaving differently by either globbing the 0xAB of the codesep as data size or not.. it really seems possible
< gmaxwell>
arubi: or not
< gmaxwell>
I think it's silly to speculate on, without testing no one really knows
< jonasschnelli>
sipa: Oh. Nice... will have a look soon!
< sipa>
oh, of course
< sipa>
arubi: i don't think that works
< gmaxwell>
arubi: e.g. perhaps any case you could construct to do that fails because earlier deserialization will fail.
< sipa>
arubi: for it to be valid post-fork, pushes/opcodes need to align with the scriptSig end
< arubi>
hm, yea.. pushdata in scriptsig without enough bytes to glob fails.. right
< sipa>
so we're perhaps back to it being a HF only under the assumption you can create a self-signing signature
< arubi>
you could with op_swap
< arubi>
or well, just op checksig right?
< arubi>
ah again, findanddelete removes it..
< sipa>
ah yes
< sipa>
doesn't findanddelete save us?
< arubi>
it almost seems like any signing shenanigans are out of the question. definitely gonna be busy thinking about this :)
< sipa>
i think just a scriptSig of "<sig> <pubkey> OP_CHECKSIG" and a scriptPubKey of "OP_TRUE" is enough to HF
< sipa>
is it not possible to create such a scriptSig now, due to FindAndDelete?
< arubi>
with FaD you'd be signing everything except the signature both pre and post fork afaict, why is this hard forking?
< arubi>
codesep is also removed, so just "<pubkey checksig 1" on both cases (codesep is to the right of checksig and removed)
< cfields>
jonasschnelli: oh! sorry, will take a look right after the meeting
< jonasschnelli>
thanks cfields
< jonasschnelli>
I think sipas 13697 should be added to the high prio list
< provoostenator>
Nice, though it would be nice if a followup PR made disable_private_keys positional, then we'll see if that part makes it into 0.17
< provoostenator>
non-positional
< jonasschnelli>
(since it's an API thing)
< sipa>
if it doesn't go into 0.17, it'll either need to be an addition rather than a replacement, or we need to mark the API experimental (which may be a good idea regardless)
< wumpus>
I think it's more relevant right now to tag things for 0.17 instead of adding them to the high-priority list
< jonasschnelli>
Or that, yes.
< wumpus>
everything that needs to go into 0.17 is high-priority by definition
< jnewbery>
in general I think it's a sensible default policy to mark new APIs as experimental
< wumpus>
(although some things are taggd 0.17 that shouldn't be,I tihnk)
< gribble>
https://github.com/bitcoin/bitcoin/issues/13426 | [bugfix] Add u8path and u8string to fix encoding issue for Windows by ken2812221 · Pull Request #13426 · bitcoin/bitcoin · GitHub
< sipa>
in some timezones it was also opened on friday the 13th
< sipa>
oh, no
< jonasschnelli>
I hope no black cat was sitting on the keyboard during coding
< wumpus>
ken2812221_: done
< ken2812221_>
thanks
< sipa>
if it is a bugfix it can also go in after the feature freeze
< wumpus>
tagged 13666 13426 and 13697 with 0.17
< wumpus>
absolutely
< sipa>
12 PRs tagged 0.17 :o
< * sipa>
fires up the review engine
< achow101>
#13712 is a bug fix for 0.17 too
< gribble>
https://github.com/bitcoin/bitcoin/issues/13712 | wallet: Fix non-determinism in ParseHDKeypath(...). Avoid using an uninitialized variable in path calculation. by practicalswift · Pull Request #13712 · bitcoin/bitcoin · GitHub
< jonasschnelli>
Looks like #8469 won't make it for 0.17. I guess untag would make sense
< cfields>
jonasschnelli: if 13617 isn't ready in time (I don't see why it wouldn't, though), we can always bump the requirement and leave the stale code for later removal
< jonasschnelli>
Oh. Right... it's not a feature... agree with cfields
< wumpus>
re-add it then?
< jonasschnelli>
Yes. I'll do it. Sry for the confusion
< wumpus>
no problem
< wumpus>
#topic exposing coin selection on RPC (achow101)
< gmaxwell>
What does that mean precisely?
< achow101>
this came up in a discussion with some companies about coin selection. basically, some are interested in using core's coin selection (or someone elses) instead of having to implement/roll their own
< wumpus>
I'd say listunspent+raw transactions API
< achow101>
currently if they wanted to use core's coin selection, the utxos need to be in the wallet, i.e. the addresses and possibly keys need to be in the wallet
< wumpus>
fundrawtransaction?
< achow101>
that is not ideal for them
< jonasschnelli>
achow101: hmm,... is RPC the right interface for that?`
< sipa>
i suggested to achow101 that a library might be a better way of doing so
< gmaxwell>
but again what does that mean. like isn't fun-draw-transaction that?
< jonasschnelli>
Right.. what wumpus said
< achow101>
the idea was then to have an rpc call where the utxos are provided and coin selection is operated on those utxos
< sipa>
gmaxwell: they don't want to use the wallet; they just want to be able to run coin selection
< wumpus>
you can import utxos into the wallet with importprunedfunds
< achow101>
wumpus: fundrawtransaction requires the wallet
< jonasschnelli>
achow101: using private key disabled dynamic wallet in conjunction with fundraw seems very efficient
< gmaxwell>
I am doubtful that it is worth our effort in maintaining a stable interface for such a thing.
< achow101>
wumpus: that's probably slow with hundreds of thousands of utxos
< wumpus>
meh, RPC is not a good place for pure utility functions
< wumpus>
if it doesn't need the wallet nor core state, why have it on *remote* procedure call?
< gmaxwell>
E.g. kallewoof's recent grouping PR would have obliterated the interface for coin selection.
< wumpus>
it just creates a central point of faliure for no good reason
< wumpus>
a library indeed sounds like a better idea
< jonasschnelli>
Agree with wumpus.
< jonasschnelli>
I know a library would be cumbersome to implement (compared to RPC),... but that is a weak argument IMO
< wumpus>
RPC is not a good way to do code sharing
< sipa>
it sounded like people really like RPC interfaces over libraries though, for reason i have difficulty comprehending
< achow101>
a library is probably better, but the issue with libraries is that they all use different languages. so a different library for each language would need to be created
< sipa>
but one possibility would be to have a separate binary with utility functions, that exposes an RPC
< gmaxwell>
its just the webby world.
< wumpus>
I don't get it
< achow101>
and they seemed to like rpcs where things are easily dropped in
< wumpus>
I don't want to debate this madness
< gmaxwell>
achow101: C(++) is callable from every language.... :P
< jonasschnelli>
Wild though: provide a cli tool?
< bitcoin-git>
[bitcoin] masonicboom opened pull request #13717: docs: Link to python style guidelines from developer notes (master...link-to-python-style-guidelines-from-dev-notes) https://github.com/bitcoin/bitcoin/pull/13717
< gmaxwell>
(well C is, C++ via making a C interface. :) )
< jonasschnelli>
*thought
< sipa>
jonasschnelli: slow
< gmaxwell>
but in any case, if you have a library you can also wrap that to present an RPC.
< jonasschnelli>
(a library & a CLI tool for the lazy people)
< wumpus>
cli also a 'remote' call, though it invokes a new process for every call!
< wumpus>
but it has the same issues with (de)serialization
< gmaxwell>
But the argument I was making above is also an argument against a library: pressure to maintain a stable interface to this would be harmful to the project.
< wumpus>
right
< jonasschnelli>
indeed... if performance is the goal, a library is probably the right choice.
< sipa>
i think coin selection is relatively well encapsulated
< sipa>
i also don't see how kallewoof's groupings would break the API
< gmaxwell>
sipa: I just gave an example where recent changes changed the API substantailly.
< gmaxwell>
Both BNB and kallewoof changed arguments to every entry point.
< wumpus>
to be honest I think this is not a concern for our project
< achow101>
gmaxwell: but the interface exposed to the callers did not
< sipa>
so? overall it takes a list of UTXOs and gives a subset back
< wumpus>
some other people want a coin selection algorithm for their own purposes
< achow101>
gmaxwell: only things within selectcoins changed, and an rpc would effectively only expose selectcoins
< wumpus>
it's fine, they can make a library out of it themselves
< gmaxwell>
it's more tightly coupled than "list of UTXOs" ... e.g. it needs fees, signature sizes.
< wumpus>
the code is open source
< gmaxwell>
I mean if someone can jigger things around to make it seperable and they find that useful, fantastic.
< achow101>
wumpus: sure the code is open source, but it is not something that can easily be taken out and dropped into something else.
< gmaxwell>
But I don't want to hear "we can't implement privacy feature X because it'll break an interface"
< wumpus>
for the consensus code I see a strong reason to provide it as a library: it is *important* that everyone uses the same consensus code
< jonasschnelli>
wumpus: I think what they want is something maintained by the same people
< wumpus>
jonasschnelli: the same people might not agree with that, who asks them?!
< gmaxwell>
perhaps they should contribute to making the wallet code better so they don't have to write their own. :P
< wumpus>
they might want the whole world to be maintained by the same people
< jonasschnelli>
heh...
< wumpus>
gmaxwell: right, exactly
< achow101>
the general feeling was that they wanted core to be more modular so that they can pick and choose specific internal components to use in their software
< achow101>
also more generally that there was some sort of "canonical" libraries for things instead of everyone implementing their own thing
< wumpus>
but making internal interfaces stable does restrict future flexibilty, as gmaxwell says
< wumpus>
it's already hard enough to change the RPC interface
< wumpus>
I understand the desire for that, bt puttingn that all on our plate is unreasonable
< wumpus>
as well as centralization
< wumpus>
it's not how things can work in a project like this, the same group providing canonical implementations for everything
< achow101>
right
< moneyball>
suggested topic: next Core dev tech meetup
< jnewbery>
wumpus: there is some benefit to making the Core coin selection algorithm more generally usable though
< wumpus>
jnewbery: people that want that, can work on that, the code is open source, they can make a library out of it
< jnewbery>
eg if we think it's good and reduces UTXO bloat, having more people using it is a benefit
< wumpus>
if it turns out to be feasible enough then core can start using that library too
< wumpus>
that's a two-step process, though
< sipa>
i think the first step for this is something we're doing anyway; making the code itself more encapsulated
< jonasschnelli>
Agree with wumpus: I guess its a one-day job to extract the coin selection code and create a library out of it...
< sipa>
no it's not
< jonasschnelli>
The burden is the maintenance...
< sipa>
it's currently split between wallet code and coinselection
< jonasschnelli>
I really like it. Will review and test it asap!
< sipa>
the other question is scantxoutset experimental or not, and with descriptor support in 0.17 or not
< jonasschnelli>
What is the difference between the FlatSigningProvider and the dummysigner?
< sipa>
dummysignatureprovider doesn't provide anything
< sipa>
flatisigningprovider takes its information from flat maps
< jonasschnelli>
I see
< wumpus>
I think scantxout should be marked experimental
< jonasschnelli>
Agree
< jonasschnelli>
Also with 13697...
< sipa>
agree
< jonasschnelli>
Helps us to do unplaned API changed
< jonasschnelli>
*changes
< wumpus>
right
< sipa>
i plan to write a longer explanation in doc/descriptors.md or so
< wumpus>
cool
< jonasschnelli>
thanks for working on this sipa
< sipa>
one of the future ideas is a PSBT standalone binary that can sign/update using descriptors
< luke-jr>
should these be a BIP? seems potentially useful outside Core
< sipa>
luke-jr: potentially yes, but not in first instance
< sipa>
i expect that this will evolve rather quickly
< luke-jr>
BIP drafts can evolve :p
< wumpus>
maybe at some point, but as this doesn't touch consensus or P2P it'd be an advisory BIP at most, and it's not required to do it first
< wumpus>
luke-jr: let's just leave it up to sipa
< sipa>
also, the part that actually needs interoperability is PSBT; descripors are just how we (or at least i hope) represent our knowledge
< sipa>
compability would certainly be useful too, but i'd rather take some time to see how things evolve
< sipa>
otherwise i fear we'll have something with a dozen optional parts all implemented by different subsets of software
< wumpus>
yes
< wumpus>
#topic bitcoin-seeder under bitcoin-core GitHub organisation (lclc)
< sipa>
lclc: no problem as far as I'm concerned, but i'm not sure it's the right message
< luke-jr>
Seems unnecessary.
< lclc>
I though because there are a few open issues and simple PRs for bitcoin-seeder and it might would make sense that several Bitcoin maintainers have merge rights.
< sipa>
there are other pieces of seeder software out there too, and having variety is a good thing here
< luke-jr>
should we put bfgminer, knots, kallewoof's script debugger stuff, etc under bitcoin-core too? :p
< lclc>
I know there are several seeder implementations but it appears to be the most used one. And since getting new node IPs by DNS is the default way to bootstrap a node in Core I think it makes sense to have one implementation in the bitcoin-core org.
< wumpus>
luke-jr: if they want
< luke-jr>
wumpus: what sipa said - it sends the wrong msg IMO
< wumpus>
yes, to be clear I don't think it's necessary
< provoostenator>
Another approach could be for sipa to give more people access to that repo?
< jonasschnelli>
No strong opinion... but slighly prefere to have it under the bitcoin-core repository
< luke-jr>
although maybe it would make sense for knots, but that would probably just be a mess on github
< luke-jr>
provoostenator: +1
< luke-jr>
to sipa giving access as desired
< sipa>
provoostenator: i'm fine with that!
< wumpus>
as I said before with the library stuff, I think it's more robust to spread projects around instead of create the illusion that they're all maintained by the same 'team'
< provoostenator>
Or create an ad hoc organization "Sipa's DNS Seed maintainer club"
< wumpus>
so agree with you on that luke-jr
< luke-jr>
there's no reason to turn access to tht eCore github repo ainto an actual position
< luke-jr>
forgive the typos
< lclc>
That's also fine with me
< achow101>
topic suggestion: moving away from bitcoin.org more
< lclc>
I (and others) just have a few simple PRs open (and open PRs feels like potential outstanding work in the future :D), so a few more maintainers (maybe jonasschnelli ?) would be good
< wumpus>
#topic moving away from bitcoin.org more (achow101)
< luke-jr>
not sure there's anything further we can do in terms of oving away..?
< achow101>
we still link to bitcoin.org for things like downloads
< wumpus>
achow101: moving what, excatly? executables have been moved
< achow101>
should probably change those
< jonasschnelli>
I'm pretty familiar with the seeder code and happy to co-maintain it
< wumpus>
achow101: where?
< luke-jr>
achow101: where?
< achow101>
in the readme
< jnewbery>
I don't think we link to bitcoin.org for downloads any more
< wumpus>
at least in the release mail I link to bitcoincore.org nowadays
< moneyball>
suggested topic: next Core dev tech meetup :)
< sipa>
For more information, as well as an immediately useable, binary version of
< provoostenator>
And there's guiconstants.h: QAPP_ORG_DOMAIN "bitcoin.org"
< jnewbery>
Open a PR to remove that link from the readme. I will happily ACK
< luke-jr>
bitcoin.org could increase distance further, but someone needs to do that work, and people whined when they tried, so I think it's stuck
< wumpus>
I think the link to bitcoin.org is only for the *general* descriptoin of bitcoin
< wumpus>
that certainly would be confusing to make bitcoincore.or
< wumpus>
moneyball: yep
< sipa>
wumpus: see quote above
< luke-jr>
provoostenator: oh yeah, that was an issue for distro stuff somewhere IIRC
< wumpus>
sipa: oh yes that should go
< sipa>
ack on that in any case
< wumpus>
provoostenator: don't change that! it determines the settings path
< luke-jr>
provoostenator: might want to make it something generic though, not Core-specific
< achow101>
I was thinking that it may not be appropriate to make release posts on bitcoin.org, at least not before they go up on bitcoincore.or
< luke-jr>
wumpus: I think it's used outside settings too, but maybe best to just leave it alone
< achow101>
as in we should simply take the bitcoin.org steps out of our release process and if someone wants to do them later, then that's fine
< luke-jr>
perhaps a comment explaining it's for compatibility, nothing more
< wumpus>
luke-jr: possibly, but at the least it loses the qsettings, that's also why it's not "Bitcoin core" named there yet
< wumpus>
achow101: would that really make anything better?
< wumpus>
achow101: are you seriously suggesting I should no longer update bitcoin.org and upload binaries there?
< achow101>
yes
< achow101>
wumpus: are you aware of the recent events on bitcoin.org?
< provoostenator>
Well, I think technically he's suggesting not having that be part of the documented process, but you can do whatever you want.
< wumpus>
acvaguely
< luke-jr>
Cobra wanted to make Knots the "default" on bitcoin.org a while back; we could encourage that, or (probably better) encourage not having a "default"
< wumpus>
so I think this will effectively reduce the number of downloads
< wumpus>
I don't care though, I don't want to be involved in political stuff at all, I' kind of burned out on that
< luke-jr>
wumpus: Core doesn't have a problem wth getting users atm
< jonasschnelli>
I think we should only publish binaries via bitcoincore.org and should not actively push it to other websites...
< luke-jr>
>99% of the network runs Core
< jonasschnelli>
It's gives a wrong sense of project coupling
< achow101>
there's ongoing work to move all of the bitcoin core stuff from bitcoin.org to bitcoincore.org
< wumpus>
maybe the developer documentation too
< luke-jr>
if people are running/updating Core simply because it's on bitcoin.org, that is kind of a systemic risk we should address if possible
< luke-jr>
(and making it harder to "just run bitcoin.org" seems a step in that direction)
< wumpus>
#topic next coredev tech meeting (moneyball)
< moneyball>
Hi, I wanted to let people know I've volunteered to organize the next Core Dev Tech meetup
< moneyball>
The current thinking is to have it in Tokyo in October after Scaling Bitcoin
< sipa>
it will in fact be my first meal of the day
< ken2812221_>
for me, it's almost breakfast.4 AM here
< cfields>
ken2812221_: you got a link to vote on the meeting time, right?
< ken2812221_>
No, I didn't receive the email.
< provoostenator>
Any tips for debugging the auto hidden service feature when it doesn't seem reachable? Its log looks happy, but trying to addnode it results in "general failure" (as opposed to "host unreachable")
< cfields>
ken2812221_: hmm. I'll resend. Can you stick around for another minute to vote?
< wumpus>
provoostenator: also run with debug=torcontrol on the server
< provoostenator>
Using the wrong port will get me a "connection refused", so something is out there :-)
< wumpus>
ok
< wumpus>
'general failure' is a generic enough error message to be completely useless :)
< cfields>
ken2812221_: please let me know when you're finished so that I can end the poll.
< wumpus>
provoostenator: so it might be something with the forwarding then, is the bind port open?
< jonasschnelli>
Is there a quick cure for my gitian builder? Trying to create a bionic base vm....but getting E: No such script: /usr/share/debootstrap/scripts/bionic
< cfields>
jonasschnelli: try using lxc
< luke-jr>
jonasschnelli: AFAIK we don't have a way to make gitian base VMs for bionic yet (at least KVM?)
< jonasschnelli>
cfields: I though I'm using lxc
< wumpus>
tor, from the server side, reports a different thing if the port is not open on the hidden service at all - versus when the connection to the local target port fails
< jonasschnelli>
Getting the error after executing bin/make-base-vm --suite bionic --arch amd64 --lxc
< cfields>
jonasschnelli: ah, I think it might still look at the bionic debootstrap script. You might need to update a package. sec.
< cfields>
jonasschnelli: oh, just apt-get update and apt-get install debootstrap
< cfields>
jonasschnelli: in my case, anyway, mine was old and missing the script for Bionic. Upgrading that package fixed it.
< wumpus>
so in any case: does everyone think I should stop uploading bitcoin core to bitcoin.org? this would be a complete break with them, maybe that's better than waiting for cobra to decide to no longer support core as he's already threatened at least once, but I'm ambivalent abouti t
< wumpus>
provoostenator: looks 100% good
< wumpus>
provoostenator: so can you connect locally to the 127.0.0.1:port that tor connects on to reach bitcoin?
< provoostenator>
Indeed. Might be nice for it to try and ping itself via Tor.
< luke-jr>
wumpus: I think we should leave that up to bitcoin.org, and encourage them to be multi-fullnode
< wumpus>
luke-jr: leave it up to them = keep doing what I do now?
< luke-jr>
wumpus: basically
< cfields>
wumpus: I think that makes sense. They're Bitcoin Core downloads, not Bitcoin downloads. If bitcoin.org wants to mirror them, that's their prerogative.
< wumpus>
ok
< jonasschnelli>
cfields: debootstrap is already the newest version (1.0.89).
< provoostenator>
bind=127.0.0.1:8333 did the trick, bid surprised that was needed.
< luke-jr>
wumpus: if they say "we'll point people to bitcoincore.org for that", then stop (and encourage this outcome)
< provoostenator>
Coming soon: I'm thinking of making my Armbian build script use the Tor hidden service stuff by default, as well as connect via Tor proxy outbound.
< ken2812221_>
The minimum debootstrap version to support bionic is 1.0.92
< cfields>
jonasschnelli: ah, you're on Debian
< wumpus>
provoostenator: yes, that's surprising, the ideal solution for that would be #8973 - make the tor forwarding code create its own private P2P listening port
< wumpus>
"I added EVHTTP_CON_OUTGOING. Making the retry work with a callback (at either http or bufferevent level) as suggested by @azat sounds like a good idea to me, but that exceeds my expertise with the libevent code."
< wumpus>
if anyone knows more about libevent than me, please help
< jonasschnelli>
cfields: so Bionic requires LXC 2.1.1 which is not available on the newest Debian version (stretch)... that seems to be unfortunate
< jonasschnelli>
Looks like I don't have an upgrade path for my build host
< wumpus>
otherwise I'm not sure this is ever going to happen, though I still think it's useful even without bitcoin-cli support
< cfields>
jonasschnelli: yea. I don't see any way around it :(
< gmaxwell>
wumpus: Do we have a way to test it without bitcoin-cli support?
< wumpus>
gmaxwell: my original PR already updated the test framework to use it (from python)
< gmaxwell>
Oh.
< gmaxwell>
Well then, I agree its useful without -cli.
< wumpus>
gmaxwell: whihch was one of the primary motivations because it allows doing the tests without claiming port ranges
< frenchy>
Hello everyone
< wumpus>
using UNIX sockets guarantees that there are no port collisions
< gmaxwell>
wumpus: yep makes perfect sense there.
< wumpus>
and reduces other risk that external things interfere ODODODwith the tests, though making the P2P port in the tests listen on localhost only was a good move in that direction
< wumpus>
frenchy: hello.
< gmaxwell>
wumpus: right, I think the reason I found the socket most interesting was better security, e.g. making it easy to give a group access to the bitcoin daemon rather than all of localhost.
< bitcoin-git>
[bitcoin] sipa opened pull request #13719: Avoid creating a temporary vector for size-prefixed elements (master...201807_psbt_no_vec_serialize) https://github.com/bitcoin/bitcoin/pull/13719
< bitcoin-git>
[bitcoin] practicalswift opened pull request #13720: build: Make lint-locale-dependence.sh work with BSD grep (avoid empty subexpressions) (master...bsd-grep-compatibility) https://github.com/bitcoin/bitcoin/pull/13720
< jonasschnelli>
cfields: self compiled lxc 2.1.1 seems to work.
< jonasschnelli>
Haven't looked at the code,... but maybe pkh(xpub68Gmy5EdvgibQVfPdqkBBCHxA5htiqg55crXYuXoQRKfDBFA1WEjWgP6LHhwBZeNK1VTsfTFUHCdrfp1bgwQ9xv5ski8PX9rL2dZXvgGDnw/1'/2/[0-1000]) should be possible
< sipa>
jonasschnelli: i would rather not do that
< jonasschnelli>
why?
< sipa>
jonasschnelli: the reason is that in some contexts where descriptors are useful, the chain size is determined by the environment (in particular, in a wallet it follows from gap limit and blockchain)
< sipa>
so unless you're going to add more "variants" of the descriptor language, it doesn't feel like a core feature
< jonasschnelli>
Thinking practical: assume I haven a xpub and I'd like to find funds via scantxoutset (gap limit not possible)... would it make sense to scan for all non hardened keys (assume BIP44 has been used)?
< sipa>
jonasschnelli: no, you specify a range
< jonasschnelli>
I'm not sure about the performance implications... maybe it's okay
< sipa>
jonasschnelli: the scantxoutset RPC in my PR lets you do that
< sipa>
it's just not part of the descriptor itself
< jonasschnelli>
aha.. okay. I'm still at the third commit. :)
< bitcoin-git>
bitcoin/master 7223263 practicalswift: wallet: Add tests for ParseHDKeypath(...)
< bitcoin-git>
bitcoin/master 27ee53c practicalswift: wallet: Add error handling. Check return value of ParseUInt32(...) in ParseHDKeypath(...).
< bitcoin-git>
bitcoin/master aba2e66 MarcoFalke: Merge #13712: wallet: Fix non-determinism in ParseHDKeypath(...). Avoid using an uninitialized variable in path calculation....
< jonasschnelli>
(sipa: IMO the third commit "Support output descriptors in scantxoutset" sould be renamed to somthing without "scantxoutset")
< sipa>
why?
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #13712: wallet: Fix non-determinism in ParseHDKeypath(...). Avoid using an uninitialized variable in path calculation. (master...check-ParseUInt32-return-value) https://github.com/bitcoin/bitcoin/pull/13712
< jonasschnelli>
sipa: it's unrelated to scantxoutset
< sipa>
the third commit is called "Output descriptors module"
< sipa>
you're looking at the PR title
< jonasschnelli>
Damit! Got fooled by github. NM
< sipa>
haha
< jonasschnelli>
sipa: right now, each child key requires to derive the complete chain, right?
< sipa>
jonasschnelli: there's a TODO to cache the bottom non-hardened key
< sipa>
i can add that in the same PR if you want
< jonasschnelli>
I guess that can be added later
< jonasschnelli>
Performance should not mater in the context of scanning the whole utxo set
< jonasschnelli>
(for other descriptor usages it may matter)
< jonasschnelli>
so later should be fine
< sipa>
right, agree
< achow101>
it would be nice if we could also get #12493 in for 0.17