< elichai2>
How stable are blocks/*.dat files? (thinking of writing a parser so you could easily query questions over all the history)
< cato_>
elichai2: what do you mean with stable?
< elichai2>
cato_: the format of the file
< cato_>
well, if you want to access data in blocks/*.dat you first need metadata from blocks/index/
< cato_>
which is a key-value leveldb. more or less you use block hashes as keys and it'll tell you where in blocks/*.dat you'll find a particular block
< yanmaani>
I have what seems to be a Heisenbug in my test; about 50% of the times I run it it gives an error. Are there any known non-deterministic footguns in the testing framework?
< wumpus>
elichai2: AFAIK the format of the block files has never been changed, there are no guarantees though, data files are not an external interface
< wumpus>
but to be honest I expect them to remain like this, it's simply a magic value then the block data, the metadata (as said) is in the database
< yanmaani>
If I have two nodes, is it always guaranteed that the blocks will propagate on time?
< yanmaani>
Or is there a way to make them ensure everything is synced?
< * yanmaani>
thanks the channel for the rubberducking
< elichai2>
wumpus: thanks. I can assume the db metadata is also quite stable? (obviously no promises)
< elichai2>
yanmaani: obv all network operations can be non-deterministic
< wumpus>
it's usually at least backward compatible
< wumpus>
yanmaani: if you have two nodes and they are connected then they should eventually sync
< yanmaani>
No, I found it
< yanmaani>
there's a specific call you can do to wait until they're all synced
< wumpus>
(there are some exceptions, e.g. a mainnet/testnet node that isn't up to date with the network won't let other nodes sync from it, but none that sould affect writing regtest tests)
< wumpus>
yes, you definitely need to wait for it, I just mean it eventually happens, there is no time guarantee
< wumpus>
fanquake: I'm going to upload rc1 executables, thanks for the reminder, I don't think this needs to be a meeting topic :)
< instagibbs>
yanmaani, there's one for blocks, one for mempool, and one for both. Typically the last is sufficient, just can make tests slower if not necessary
< instagibbs>
pretty much any time your test "alternates" between two nodes it can be necessary for non-flakey behavior
< somethingsomethi>
#proposedmeetingtopic taproot in 0.20
< bitcoin-git>
[bitcoin] jonatack closed pull request #19405: rpc, cli: add network in/out connections to `getnetworkinfo` and `-getinfo` (master...in-and-out-connections) https://github.com/bitcoin/bitcoin/pull/19405
< bitcoin-git>
[bitcoin] jonatack closed pull request #19455: rpc generate: print useful help and error message (master...rpc-generate-helpful-error_message) https://github.com/bitcoin/bitcoin/pull/19455
< bitcoin-git>
[bitcoin] jonatack closed pull request #19397: test: resyncing to tip of blocks generated after invalidateblock (master...p2p-invalidateblock-tests) https://github.com/bitcoin/bitcoin/pull/19397
< jonatack>
Cleaning out stuff with no ACKs to make room for the new.
< wumpus>
jonatack: fwiw I just didn't get around to reviewing #19405 again in detail yet, I'm definitely not against it
< gribble>
https://github.com/bitcoin/bitcoin/issues/19405 | rpc, cli: add network in/out connections to `getnetworkinfo` and `-getinfo` by jonatack · Pull Request #19405 · bitcoin/bitcoin · GitHub
< wumpus>
of course it's up to you if you close your PRs or not, but it might make sense to ask for more review instead, things have just been a bit slow for a while with regard to review compared to the number of PRs submitted
< jonatack>
wumpus: thanks, there are more interesting things i want to work on and don't want to encumber the too-high PR stack with proposals that aren't "hell yes" or have more than a handful of PRs open
< jonatack>
also, i think modding your script into a custom dashboard i'm using to observe connections in much more detail by type and eviction criteria made just having in/out seem a little pale :)
< wumpus>
Ok :)
< fjahr>
jonatack: maybe 19405 can still be marked as "up for grabs"?
< jonatack>
fjahr: sgtm
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu Jul 23 19:00:20 2020 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< wumpus>
jeremyrubin: well if it is a blocker for you it should be added as blocker, if not not, it's not like everyone needs to have a blocker slotted every week :)
< jnewbery>
I'd like to make some progress on #19398. The first PR in that sequence is #19472 so perhaps that could be added to high priority?
< jeremyrubin>
There's like 70 commits worth of other work that is queued up pending on progress on similar stuff for like the last 6-9 months
< sipa>
jeremyrubin: ok, so it's a blocker; no other justification needed :)
< jeremyrubin>
Those could be all parallelized I guess, but I don't want to overwhelm review
< wumpus>
jeremyrubin: jnewbery added
< jnewbery>
wumpus: dank
< jeremyrubin>
tx
< wumpus>
#topic taproot in 0.20 (somethingsomethi)
< somethingsomethi>
Like I said, this came up in the taproot-activation channel: When could we really start an activation cycle?
< somethingsomethi>
Going by the segwit playbook, the activation params would be released in 0.21.1, which is probably about 8 months from now, so some people were wondering if this could be shortened
< wumpus>
welcome btw somethingsomethi I don't remember seeing you in a meeting before
< somethingsomethi>
yeah, made this account just two hours ago :-)
< somethingsomethi>
I'm really more trying to pass messages between irc channels
< luke-jr>
I was expecting a 0.20.2 with it, but apparently there's a dependency on wtxid relay?
< wumpus>
I think the question of whether to backport it to 0.20.x is completely independent of shortening anything
< somethingsomethi>
for the purposes of the taproot-activation channel, it would be interesting to know how much time there is before anything starts
< jeremyrubin>
I think there's sort of a tradeoff; how incompatible is master with 0.20 right now v.s. how incompatible will 0.21 be with 0.20?
< luke-jr>
wumpus: well, if it didn't have the wtxid dependency, we could release 0.20.2 in a few weeks with it…
< wumpus>
I think the expectation is to backport it to that branch when it's ready
< luke-jr>
(assuming the usual requirements get met quickly)
< instagibbs>
wumpus, that meaning 0.20(handwaving away deps)
< instagibbs>
?
< jeremyrubin>
wumpus: or maybe backport it to the current major release/last major release when it's ready
< instagibbs>
or whatever branch is ready by then
< wumpus>
jeremyrubin: yes
< jeremyrubin>
+1, here's to hoping that does mean 0.20 tho
< wumpus>
in any case it shouldn't end up in a .0 first
< sipa>
i think it's very reasonable that the consensus logic for it goes into a .0
< instagibbs>
activation I presume he meant
< sipa>
and then activation goes into a .1
< luke-jr>
I suppose another option would be to just start the 0.21.0 release process after 0.20.1 gets finalised, so that 0.21.1 could be sooner
< wumpus>
I mean the actual activation
< sipa>
the real question here is when the consensus logic can be in a release, as that's a lower bound
< wumpus>
of course ,preparations can go in sooner
< jeremyrubin>
I'm somewhat of the opinion that if a release cycle schedule is holding up when we think a consensus thing gets put out then it makes sense to shift around the release windows, so i agree with luke-jr
< wumpus>
but is this really a problem right now?
< jeremyrubin>
mac recently released an OS with two version numbers btw so that, e.g., 20.2 is identical to 21.1
< sipa>
if there is an expectation to backport the consensus logic to a 0.20 branch, we could take some care to avoid having too many preparation changes in master first
< jeremyrubin>
wumpus: in the ##taproot-activation discussion, yes
< wumpus>
I'd prefer not to shift along major version schedules too much
< luke-jr>
wumpus: I think generally there's a sense that we should have at least a 1 year timeout, and waiting a year before even starting that annoys people :p
< sipa>
i wasn't expecting taproot in a 0.20 branch- but i also sympethize with the argument that consensus changes, when reviewed and agreed upon, shouldn't be too strongly affected by bitcoin core's release schedule
< wumpus>
in any case, the release schedule for 0.21 is in #18947, is this a problem? do you think it should be moved backward or forward?
< luke-jr>
wumpus: the release schedule isn't a matter of quality, though, is it?
< sipa>
luke-jr: agree, but it's a matter of development cadence
< wumpus>
I'm fairly sure a predictable schedule helps there
< wumpus>
if people on twitter don't understand the release schedule, explain it to them
< jeremyrubin>
i tried and they told me that I'm trying to attack bitcoin :p
< luke-jr>
I suppose I could put it in Knots..
< wumpus>
sigh
< somethingsomethi>
so 0.21.1 around feb it is then?
< luke-jr>
was planning to wait in case the protocol has any further changes though
< wumpus>
that sounds like just riling up things
< luke-jr>
wumpus: ?
< sipa>
if there is a strong desire to have taproot earlier in a bitcoin core release, i think we should prefer backporting to 0.20 over accelerating 0.21
< wumpus>
"that I"m trying to attack bitcoin" I mean
< luke-jr>
ah
< instagibbs>
sipa, ok that's fair enough
< sipa>
(whether that desire exists, i have no opinion about)
< wumpus>
there's lots of trolls on twitter we know that
< luke-jr>
sipa: so two backports?
< luke-jr>
sipa: one with wtxid relay, then the next with Taproot?
< sipa>
luke-jr: i don't have a good answer here -i feel that wtxid deployment will take way longer than people here seem to tolerate for taproot
< sipa>
regardless of what releases it goes into
< sipa>
as it takes sometimes ~years to get new releases deployed...
< jeremyrubin>
it also matches roconnor's complaints
< luke-jr>
well, at least then we can tell people "not enough wtxid relay users yet" if they whine about the delay? :p
< jeremyrubin>
people should care much more about wtxid relay than taproot in some respects
< sipa>
wtxid doesn't really hurt until there are major relay policy changes
< sipa>
s/hurt/help/
< wumpus>
this is the first time I hear about wtxid relay being a problem for taproot
< wumpus>
if I knew this sooner I would not have merged it
< ariard>
we talked about it like 2 or 3 meeting away
< luke-jr>
wumpus: the lack of it is the problem..
< ariard>
when moneyball brought taproot topic
< wumpus>
we can still revert that PR I suppose though it sounds like a mess
< sipa>
wumpus: the opposite; wtxid is arguably required to be deployed on the network, before we can make major relay policy changes
< sipa>
(such as taproot would imply)
< wumpus>
ok
< luke-jr>
sdaftuar: is wtxid relay protocol stable? should we update the BIP to Proposed status?
< sipa>
or any script extension on top of segwit
< ariard>
luke-jr: it has been merged yesterday, sipa has some follow-ups needed
< sipa>
the follow-ups don't change the (specified part of) protocol
< wumpus>
so at least merging it is progress right?
< sipa>
yes, definitely
< jeremyrubin>
I think the problem is that people are proposing a 4 year rollout timeline for the soft fork, to be conservative. If we're not releasing any clients for that until february it's like 5 years. I think we're inviting controversy if we do that.
< somethingsomethi>
any guesstimate on how long the gap between wtxid and taproot would have to be?
< luke-jr>
jeremyrubin: ?
< wumpus>
february?
< luke-jr>
jeremyrubin: I don't see the community tolerating 4 years
< somethingsomethi>
*wtxid and taproot rollout
< jeremyrubin>
"proposed 0.21.1 release"
< luke-jr>
somethingsomethi: depends on how long it takes nodes to upgrade
< somethingsomethi>
luke-jr that's why I wrote guesstimate :-)
< jeremyrubin>
I don't see it being tolerated either, but I think that's what a decent number of people are expecting and proposing.
< jeremyrubin>
By being slow we end up inviting a big kerfuffle UASF thing again.
< wumpus>
I really dislike this discussion about tolerating, things are ready when they're ready
< wumpus>
it sounds like you're trying to pressure us and I really dislike this
< ariard>
You can build most of applications improved by schorr/taproot today (in hacky way though)
< jeremyrubin>
you can draw a line between what I'm expressing and what I'm relaying
< somethingsomethi>
It's not so much about pressure, if everyone agrees it's at least a year out, then at least we can plan accordingly
< ariard>
like ecdsa scriptless scripts
< instagibbs>
ariard, i.e., probably not at all :)
< jeremyrubin>
sorry if it's coming across as *me* pressuring
< jeremyrubin>
I agree with the "release when it's ready" mentality
< wumpus>
if you want to push this I'm going to quit as maintainer
< wumpus>
I feel enough responsibility for this to not like if things like this get rushed
< sipa>
fwiw, i don't think there is much in master today that interacts with taproot, that would be hard to backport
< sipa>
i think the bigger question may be wtxid relay
< ariard>
instagibbs: I disagree a bit, some people instead of complaining are building on ecdsa while envisioning to upgrade to schnorr
< luke-jr>
I guess I can look into backporting that
< ariard>
for their use-case, when ready
< aj>
sipa: updating secp in a backport would be large, but self-contained, so preumably fine?
< jeremyrubin>
sipa: I think we can reasonably release taproot without wtxid relay, no? Taproot blocksonly validation, don't accept things to mempool?
< instagibbs>
fantastic ariard
< wumpus>
FWIW I like taproot but this is not some shitcoin where we rush features in to boost the price or something like that
< sipa>
jeremyrubin: ugh...
< ariard>
jeremryrubin: hmmm wouldn't this break compactblock?
< sipa>
aj: yes
< luke-jr>
ariard: no?
< instagibbs>
sipa, ok so you said it takes time for wtxid relay to propagate, how much time? Is this a hard blocker from even setting a signaling window?
< jeremyrubin>
sipa: I mean, whatever is less software work to get it so that clients can take a backport...
< luke-jr>
instagibbs: IMO we shouldn't set a signaling window until at the very least Taproot is merged into master..
< ariard>
luke-jr: I mean loosing the propagation benefits of validating transaction before seeing them in a block?
< instagibbs>
luke-jr, right I assumed that
< instagibbs>
I'm asking *bare* minimum
< instagibbs>
obviously things take time on top
< luke-jr>
ariard: sure, but that's true even without Taproot logic
< jeremyrubin>
ariard: the idea would be that wtxid can't be backported, so it's in 0.21. 0.20 gets degraded compactblock performance
< jeremyrubin>
luke-jr: good point, all old clients get degraded compactblock
< sipa>
instagibbs: i need to think about that
< luke-jr>
it would be interesting to split policy acceptance of Taproot, from Taproot consensus deployment
< instagibbs>
sipa, roger
< cfields_>
jeremyrubin: that sounds interesting to me. sipa: why "Ugh" to that?
< ariard>
that sounds a bit awful for network robustness, you will favor connections to upgraded nodes
< luke-jr>
ariard: there is no uniform network policy anyway, nor is the network dependent on that
< sipa>
well, the network won't actually have any taproot-spending transactions until activation, so i think anything before activation doesn't matter
< jeremyrubin>
instagibbs: an interesting corollary of your point is it may be good to try to backport wtxid relay even earlier (e.g., 0.19 if possible) to boost those #'s
< instagibbs>
that might be too difficult/dangerous
< instagibbs>
(I dont know)
< luke-jr>
well, if we split Taproot consensus vs policy… we only need wtxid relay for policy, right?
< ariard>
luke-jr: I was thinking consequences on peer eviction/selection, if you degrade performances of a whole subset of them
< aj>
there's been a punch of p2p changes since 0.20 that will probably make backporting wtxid relay a bit hard at this point (not very hard if you just rewrite it; pretty annoying if you try cherry-picking/rebasing)
< luke-jr>
so that can proceed in parallel to the consensus/activation
< sipa>
luke-jr: i don't see how that is related to splitting anything
< luke-jr>
if activation happens ~a year from now, having only consensus logic in 0.20, and wtxid relay + Taproot policy only with 0.21+ seems fine?
< sipa>
relay of those relevant transactions won't happen until activation
< luke-jr>
sipa: right, which is why we don't need to sort relay issues until closer to activation
< sipa>
yes, i agree with that
< luke-jr>
we could start the Taproot activation process, and even in the worst case scenario (too few updated to 0.21) have it activate without changing the node policy
< instagibbs>
mmmm
< sipa>
wumpus: do we have any other topics?
< sipa>
i feel the part of the discussion relevant to bitcoin core's release process is over
< wumpus>
no, that was all
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Jul 23 19:49:37 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< somethingsomethi>
ok, so bottom line: There are a couple of interesting ideas, but probably it'll be 0.21.1, and relay might come some time after that. Is that about right?
< sipa>
relay after that? what?
< aj>
wtxid relay is already merged in master, so will be in 0.21.0
< aj>
(unless there's some horrible bug or something)
< jeremyrubin>
I think luke-jr is going to try to summarize in ##taproot-activation
< instagibbs>
I think a summary would be great, I think I grasped the main points at least
< instagibbs>
thanks for input sipa wumpus
< luke-jr>
yeah, I dropped 3 lines; feel free to add more if I forgot anything
< luke-jr>
somethingsomethi: the split of consensus/relay is only needed if we do deployment of Taproot on 0.20
< ariard>
for folks working on p2p, what's your opinon between transaction request overhaul or erlay getting first ?
< somethingsomethi>
sipa: I mean the first step could be just to activate the validation parts, but not relay transactions until wtxid is sufficiently widespread
< ariard>
(a gleb question who is away to decide on rebase order)
< sipa>
somethingsomethi: then what's the point?
< sipa>
"taproot is active, but you can't actually spend anything yet!"
< somethingsomethi>
luke-jr ah ok. I though it might be an option either way if wtixd uptake is too slow
< aj>
ariard: overhaul first was my thought
< luke-jr>
sipa: the point is to have the consensus side ready when the rest is
< luke-jr>
sipa: since the consensus side may have a long simply-waiting period for activation
< jeremyrubin>
realistically it will be like a year after activation before there's any big uptake on taproot anyways
< sipa>
sure, but having it active before it can be used due to relay reasons is pointless
< somethingsomethi>
sipa: Well, sure. You could look at it as a way to de-risk deployment, but yeah, it wouldn't be overly useful
< somethingsomethi>
at least the second part wouldn't require so much coordination though
< jeremyrubin>
clients can reject invalid spending blocks which is super useful for deployed infra
< luke-jr>
sipa: hopefully 0.21 will be widespread enough before the actual activation occurs
< instagibbs>
It could theoretically happen if phased consensus->relay deployment had the 2nd part delayed, so important to think about? Meanwhile people can still uptake
< sipa>
aj, ariard: if erlay is done without the 128-bit txids (which perhaps it should), i think it's almost orthogonal to the tx request overhaul
< sipa>
as erlay would reuse inv/getdata for the final stage in that setup
< aj>
sipa: oh no, are you attacking my excuse for not reviewing erlay yet?
< sipa>
aj: thanks for reminding _me_ i don't have an excuse not to review it!
< ariard>
I think the last erlay version doesn't have 128-bit txids and new invtx/gettx compare to bip
< instagibbs>
aj, "I'm lazy" <--- good enough
< instagibbs>
ariard, you're correct(at least last time i talked to gleb)