< sipa>
i really don't understand what drives people to open issues with random words
< sipa>
i can understand someone spamming a bitcoin address, or people confusing pull request to be something they need to get the source code... but this?
< fanquake>
Yea it’s just nonsense
< sipa>
"geld" is dutch for "money" fwiw
< fanquake>
Well, I guess that does make slightly more sense heh
< sipa>
slightly
< the_fly_>
hi all, is it possible to output all block hashses (and rawtx) starting from the genesis block from via ZMQ?
< the_fly_>
in the past i had forked bitcoind and added zmq support myself, upon rescan i was able to do this
< sipa>
much better to use the rest interface for that
< the_fly_>
i was considering using the RPC interface but i would like this to keep synced with live transactions after it has reached the top of the chain
< the_fly_>
and i expect scanning over all blocks and TXs will be quite slow
< MarcoFalke>
I think the GitHub App rewards people for any kind of interaction, thus incentivises spamming
< sipa>
the_fly_: rest, not rpc, but ok
< the_fly_>
ah sorry, i am unaware of the rest interface
< MarcoFalke>
There is also -blocknotify
< sipa>
it's very similar but is just queried by URL; no need for RPC
< sipa>
the_fly_: in any case, no, ZMQ is only intended for notifications, not historical querying
< the_fly_>
MarcoFalke: does blocknotify impact the behavior of the ZMQ interface?
< MarcoFalke>
it shouldn't
< sipa>
it's not very appropriate for thag either, as ZMQ has no reliability guarantees
< the_fly_>
yes, i actually bridged it with rabbitmq, so at least i could keep a persistent queue
< sipa>
generally the suggested approach is having something that polls for updates, and has a fast path to trigger looking on zmq/blocknotify/...
< the_fly_>
i can probably live with blocknotify, and then query (rest or rpc) the raw tx info for each tx in the block
< the_fly_>
so no need for rescan, just getblockhash until i reach the end of the blockchain, but whilst that's happening i can populate blocks which are mined
< the_fly_>
i can live with some inefficiency if scanning the entire block & transaction history is tractable within some reasonable timeframe (days)
< the_fly_>
the end goal is to populate a database so i can perform various queries and visualizations of the blockchain
< sipa>
there are existing projects that dobthat
< the_fly_>
when i implemented my solution initially there wasn't anything, it worked but i would rather just use stable bitcoind than a fork
< the_fly_>
the nice thing about blocknotify based solution is that you are in sync with the state of your full node
< the_fly_>
iirc the zmq support i hacked in would spit out all blocks during a rescan
< the_fly_>
so you could nuke the database and sync all blocks, or and catch up in the case of any network outtage etc.
< the_fly_>
this time around i need to query transactions, and store their fields in a structured way (likely just postgres columns)
< the_fly_>
a bespoke solution was more efficient because i could have more control over when indexes were created (is faster to populate first, then create the indexes)
< the_fly_>
oh right, thanks, i think i should be fine with the current rpc+zmq support, i was just hoping there was a way to avoid querying every block+tx for the initial bootstrap
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #20908: fuzz: Use mocktime in process_message* fuzz targets (master...2101-fuzzMocktime) https://github.com/bitcoin/bitcoin/pull/20908
< aj>
oh libs have to be in order in general apparently? geez. TIL apparently?
< sipa>
aj: yes, dependency order, unless they"re grouped, in which case the linker loops
< vasild>
jnewbery: aj: I guess the benefit of having a separate classes for peer manager "implementation" and "interface" is not clear from 20811 alone.
< vasild>
yes, this is why I say "from 20811 alone" :)
< jnewbery>
ah ok
< vasild>
I went looking for "why do that?" answers in the other PR :)
< jnewbery>
It also helps with https://github.com/bitcoin/bitcoin/issues/19398, which is gradually moving data into the Peer struct. Having that only declared internally in net_processing.cpp reduces header churn
< wumpus>
aj: linker order is important with static libraries; dynamic libraries contain the list of NEEDED libraries they depend on in turn so there it doesn't matter
< wumpus>
would be nice to get some more review on #18710, it's fairly straightforward but as it affects the thread pool used for script verification, which is consensus code, it needs some more eyes on it
< aj>
wumpus: yay! / oh no! now i need another PR for the typo things i forgot from there
< wumpus>
MarcoFalke: not that i know of, unfortunately
< aj>
"move away from github" is the answer for everything involving loose commits, isn't it?
< wumpus>
aj: the help/manpage still needs to be done anyway
< aj>
wumpus: oh bitcoin-util needs a zillion things, i just meant some of the signet readme bits
< wumpus>
aj: right
< MarcoFalke>
wumpus: Any thoughts on radicle? Havne't tried it, but it makes issues harder to report for one-offs, right?
< fanquake>
wumpus can you block bitmastercoin shortly
< MarcoFalke>
Also, radicle will probably break email notifications (one of the few nice things that GitHub offers)
< wumpus>
MarcoFalke: i like the direction of that project, and i think the design is pretty elegant; it still needs a lot of functionality to be able to replace our use of github, though
< fanquake>
wumpus: ok. If you can’t find any explanation tonight, I’ll follow up with GH
< wumpus>
it is the same for other organizations
< wumpus>
checked on another computer, same... can oneone who is owner of a gh organization please check https://github.com/organizations/<org>/settings/user_blocks ?
< jnewbery>
Before we get on to those, I suggest we start with our regular topic of priorities. What are people working on/hoping to make progress on over the next weeks/months?
< jonatack>
#20197 and adding more unit test coverage to the eviction logic
< jnewbery>
My main priority for the start of this year is to make progress on the net/net_processing split (#19398). I'm currently waiting for #20811 because I think my remaining changes are less disruptive after that.
< sdaftuar>
it's gotten a decent bit of feedback so far, so not sure if it's helpful for me to go through all the context again?
< sdaftuar>
but i can for anyone who hasn't looked at it
< sdaftuar>
the main goal here is to be able to increase the number of inbound connection slots on the network, in order to feel good about increasing hte number of block-relay-only connections we make
< sdaftuar>
which in turn is in order to add security to the network by increasing its partition resistance, at relatively low cost (block-relay-only connections are low-resource)
< sdaftuar>
but to do all that, we need to give nodes a way to know that an inbound peer is a block-relay-only peer. currently block-relay-only peers use the fRelay flag from BIP37 to instruct their peer they don't want transactions
< sdaftuar>
but BIP37 allows for transaction relay to resume with a FILTERCLEAR message
< jnewbery>
sdaftuar: when you say low-resource, are you mostly talking about memory, bandwidth of something else?
< jnewbery>
*or something else
< sdaftuar>
memory and bandwidth
< sdaftuar>
and cpu, i guess
< sdaftuar>
so my proposal is to add a new p2p message ("disabletx") which means that a connection will never relay transactions, for its lifetime
< amiti>
sdaftuar: do I understand correctly the current proposal is a `disabletx` messages that explicitly disables transactions & implicitly disables addresses? And the thinking is that in the future we could have a message that explicitly enables addresses ? I followed your reasoning for opt out vs opt in (why would we ever opt out of blocks?), but what do you think about implementing block-relay conns with two
< amiti>
messages, `disabletx` and `disableaddr` or something?
< vasild>
if the node is up to date, then block-relay conveys just a few MB of data per 10 minutes, right? But if the node is not up to date...
< sdaftuar>
this in turn will allow us to write code to increase the number of connections lots by reserving additional slots for disabletx-peers
< sdaftuar>
amiti: right, so block-relay-only connections currently have no way to communicate that they also don't want addr messages
< sdaftuar>
and my proposal for now is to just have the BIP "RECOMMEND" that we not send addrs to peers sending disabletx
< sdaftuar>
with the idea being that in the future, we should adopt some kind of addr relay negotiation protocol, which would then take precedence
< sdaftuar>
however i think designing an addr relay protocol will take some work, and i'm not ready to propose one now
< sdaftuar>
i think there are a bunch of questions around what the goals of addr relay should be, and how best we might achieve them
< sdaftuar>
and so i don't think it makes sense to propose something that we'd likely just want to change soon after
< amiti>
ah I see, you think the long term design should be more than a toggle (like with txs)?
< sdaftuar>
yeah, at the least i suspect we want to communicate information about particular networks (as described in BIP 155)
< amiti>
yeah, that makes sense.
< sdaftuar>
and it's not clear to me yet what our different handling should be for different networks (ie addresses we understand versus ones we don't)
< vasild>
tx relay is unrelated to addr relay, linking both together under "disabletx" would be confusing?
< sdaftuar>
vasild: both are related to block-relay-only peer logic, which is the only thing that will be using this at the start... i don't see why it's a problem to have a recommendation for this?
< sdaftuar>
software that ignores the addr-relay suggestion will not be in violation of the design
< sdaftuar>
but it accommodates our desired behavior today
< sdaftuar>
if there were some other software that was going to be using this which wanted different functionality, then i might agree
< jnewbery>
I agree that it seems strange to link the two things (tx relay and addr relay) in the BIP
< vasild>
then isn't s/disabletx/relayblocksonly/ more clear (relay only blocks and nothing else - no tx, no addr)?
< ariard>
but the alternative to introduce a `disabeladdr` will be superceded when we have a better addr relay protocol?
< sdaftuar>
vasild: that was my first approach, but many people seemed to feel that having a BIP that governed too much behavior was also confusing or bad protocol design, for future compatibility
< vasild>
:)
< sdaftuar>
vasild: so this approach is narrow -- just specifies required behavior for tx-relay -- but the motivation gives us an implication for other behavior as well
< sdaftuar>
in the absence of protocol support
< jonatack>
sdaftuar: fwiw i like the original "blockrelay"
< sdaftuar>
(trying to figure out how to do an ascii shrug)
< jonatack>
sometimes in those discussions the squeaky wheel gets the grease, so mentioning it as a fwiw here
< sdaftuar>
i mean i could go either way. i think there's benefit to understanding the network from communicating more exactly what these connections do, and establishing first-class support in the protocol
< sdaftuar>
which was more inline with my initial proposal / description
< sdaftuar>
but i can see the argument that it imposes design burden on future protocol implementers to figure out how new features should interact with this
< sdaftuar>
so making this more narrowly tailored shifts the burden back on just software implementing this feature
< jonatack>
jnewbery: nvm, GitHub doesn't work for me correctly, your link sent me to a different one
< sdaftuar>
i think i can implement what i would like to implement with either approach, so i'm hoping that we can just pick a path and move forward.
< sdaftuar>
i don't think my current proposal inhibits any future protocol extensions at all
< jnewbery>
I don't see much difference between the original proposal and the disabletx proposal. They both interact with addr relay in a potentially unexpected way.
< sdaftuar>
jnewbery: but it's just a recommendation. what drawback is there to having it in there?
< sdaftuar>
obviously it could be removed; it just seems to me that it makes things strictly worse.
< jonatack>
sdaftuar: hm, the latest push of 20726 seems smaller and inbound-block-relay was dropped?
< ariard>
sdaftuar: implementation-wise, how do we tread `disabletx` peers keep sending us addrs? we disconnect or ignore msgs ?
< sdaftuar>
jonatack: yes, i switched to using just a bool to track whether the peer is a disabletx bool, rather than upgrading connection type.
< jnewbery>
If there is some new addr relay negotiation method introduced in future you have to specify how that interacts with the disabletx BIP
< sdaftuar>
jnewbery: obviously it takes precedence, as the disabletx BIP only has a recommendation, and not MUST semantics, and it even calls out this possibility
< sipa>
good morning
< jonatack>
sdaftuar: ok. we already have a de facto inbound block relay type IIUC and it would be nice to formalize it, if so, but I understand that it may not be central to your proposal
< sdaftuar>
jonatack: there was feedback that the connection type change was not the right fit for this. i don't feel strongly, but in changing the thinking from explicitly communicating BLOCKRELAY to a peer, to sending a DISABLETX, it felt more in line with just another setting on a peer's preferred transaction relay
< jnewbery>
I think it's probably more useful to discuss the protocol proposal at this stage than implementation details
< sdaftuar>
here's my question for people who are concerned abotu the addr relay interaction:
< sdaftuar>
let's say we deployed disabletx. shoudl bitcoin core relay addrs to software sending these messages?
< vasild>
yes
< amiti>
I would prefer if there was a way to be explicit about what we are relaying (tx / block / addr). both the original & current proposal are suggesting one message to communicate two pieces of information. I see the challenges of designing for a future where we want more advanced addr relay, but I'm still wondering how we could do both.
< sdaftuar>
vasild: the fact that we relay addrs to inbound-block-relay-only peers is a problem, no?
< amiti>
hm, if bitcoin core relayed addrs to software that sends disabletx, we aren't able to take advantage of the savings that is the goal, right?
< sdaftuar>
isn't that behavior strictly worse than not relaying addrs?
< sdaftuar>
amiti: that's a minor goal. addr relay is not nearly as big a deal as tx-relay
< sdaftuar>
the data structure is much smaller, i believe
< amiti>
ok
< sdaftuar>
but yes, a bit annoying
< ariard>
it can be cleanup when an addr relay negotiation protocol is introduced
< sdaftuar>
one of the drawbacks to increasing the number of inbound-block-relay-only peers is that we create a bunch of addr-relay blakc holes
< sdaftuar>
i don't see why we can't make progress on this while deferring design of an actual addr relay protocol ot the future?
< jnewbery>
amiti: the receiving node could just drop all addr/getaddr messages. There wouldn't be any memory cost and minimal bandwidth cost
< vasild>
I mean "yes", because they did not send "disableaddr" (which does not exist) -- i.e. don't link addr and tx relay in one option
< amiti>
jnewbery: ah. right.
< jonatack>
sdaftuar: ack
< jnewbery>
sdaftuar: do you have any further thoughts about what a future addr relay negotiation method would look like? I know you were pushing for something similar with BIP155
< sdaftuar>
i have a guess, but i think we need to get everyone on the same page about the goals of addr relay are, and come up with some relay policies that we think would achieve that goal, and then make sure our design supports those relay policies
< sdaftuar>
there's currently no writeup anywhere on how addr relay should work, to my knowledge
< sdaftuar>
but my naive guess is that adding some kind of feature negotiation where we signal support (not sure 0/1 or if more precision is necessary) for different networks (as defined in bip 155) is probably what we will eventually want
< sdaftuar>
i just can't defend that right now
< jnewbery>
what's the downside of removing the addr relay RECOMMEND point and punting on how to do addr relay negotiation?
< sdaftuar>
jnewbery: first you answer my question from above please!
< ariard>
even assuming you have one goal, and nodes operators might wish different relay strategies in function of their privacy needs
< sdaftuar>
let's say we deployed disabletx. shoudl bitcoin core relay addrs to software sending these messages?
< sdaftuar>
i think the answer to that is of course not
< jnewbery>
it depends what the spec says, of course
< sdaftuar>
well then that creates the black-hole problem, which will prevent us from wanting to add more blokc-relay connections
< sdaftuar>
i mean we can certainly gate all these improvements on addr relay improvements. i just think that will set us back a long time
< ariard>
sdaftuar: to be clear you're envisioning 1) adding more block-relay connection and then 2) addr relay improvements?
< sdaftuar>
ariard: yes i think we know how to do 1), while 2) requires research
< jnewbery>
You don't want a disableaddr message because it won't be useful after the new addr relay negotiation method?
< sdaftuar>
i've started doing some research on 2), but it might be a year or more before i feel good about proposing anything, i don't know
< sipa>
sdaftuar: i hadn't considered the fact that adding more block-only connections (a goal of disabletx), actually worsens the addr black hole problem
< jonatack>
sdaftuar: i agree that incremental and possibly iterative is the pragmatic way forward here
< sdaftuar>
sipa: this came up in #15759 when first proposed, and was one of the reasons we kept it at 2 connections
< sdaftuar>
based on looking at my own nodes at the time, i didn't feel it was a material worsening of the sitaution, given the other software on the network
< sdaftuar>
but i would feel uncomfortable advocating going up to 8 peers without really looking at it a lot more carefully
< sdaftuar>
in the absence of us changing our default behavior for disabletx peers
< sdaftuar>
jnewbery: yes i think a disableaddr message would be short-lived
< sipa>
right
< ariard>
to be back on the question "should bitcoin core relay addrs to software sending these messages?" _currently_ we don't accept addr msgs from outbound-block-relay peers so why it's worsening the black-hole problem?
< sdaftuar>
ariard: the outbound peer will choose us for addr relay (sometimes)
< sdaftuar>
and we will drop it and not relay further
< sdaftuar>
so increasing the number of connections that drop addr messages could inhibit addr relay
< sdaftuar>
this gets to what i was saying before though, i don't think anyone has studied addr relay. this is an open area of research that is (to my knowledge) very undeveloped
< ariard>
yeah because you have less peers doing useful addr relay, but I think we should defer this discussion when we introduce more block-relay connectiions
< sdaftuar>
so i'm just trying to not make things worse
< sdaftuar>
ariard: well i'd like to establish that people are comfortable with not relaying addrs to disabletx peers
< sdaftuar>
because if not, then i don't to advocate for this
< ariard>
sdaftuar: and my feeling is some folks would prefer introduce a `disableaddr` to have a clean protocol signaling
< sdaftuar>
(not relaying addrs to disabletx peers until such time that we deploy addr-relay negotiation, to be clear)
< ariard>
I'm ~0 on this
< sipa>
i don't think it's unreasonable to recommend not relaying addresses to disabletx peers; so far addr relay is completely local policy anyway with no BIPs discussing it at all afaik
< sdaftuar>
sipa: my understanding as well that there are no BIPs on it
< sipa>
it's what we'd do anyway, unless the BIP explicitly says the opposite
< jnewbery>
any additional points, or should we move on?
< jonatack>
subject to review, concept ack on the BIP draft and 20726
< jnewbery>
jonatack: we only have 5 minutes left, so I propose we punt your topics to the next meeting
< sdaftuar>
thanks. i'm planning to mark 20726 ready for review once a bip number is assigned, fyi
< jnewbery>
One quick topic before we end. How do people feel about a different time for this meeting? It would have been nice to have aj here, but it's 1am for him, so it's not the most sociable time.
< jonatack>
jnewbery: that's fine. for now i'd only like to mention that connection type implementations are open for the CLI and GUI (and partially merged in the GUI), in case people didn't see them
< jonatack>
and they have had to go beyond the literal connection types to cover the actual cases
< jnewbery>
2100 UTC is 07:00 in Brisbane, 16:00 in New York and 13:00 in California
< sipa>
jnewbery: earlier will be hard for me
< vasild>
21:00 UTC is 22:00 central europe, I will not be able to attend
< sipa>
timezones suck :(
< vasild>
(I rarely attend the other two meetings because they are too late too)
< jonatack>
agreed
< jonatack>
(they suck and the meetings are late :)
< jnewbery>
ok, let's keep it at the current time for now.
< MarcoFalke>
If sipa moves to the east coast, all our problems are solved, no?
< sipa>
nobody else on the west coast?
< sipa>
if so, there is a 9 hour timezone gap between US east coast and eastern australia
< MarcoFalke>
19:00 au, 11:00 europe, 7:00 east coast?
< jonatack>
1100 CET is 0600 EST
< jonatack>
er, 0500 EST
< jonatack>
6 hours
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #20915: fuzz: Fail if message type is not fuzzed (master...2101-fuzzFailMsgType) https://github.com/bitcoin/bitcoin/pull/20915
< MarcoFalke>
There is also +-2 hours due to DST :(
< jeremyrubin>
it looks to me that if the legacy provider is used then the data just gets dropped instead of written into the psbt
< sipa>
jeremyrubin: that does look weird... but it's the same in DescriptorScriptPubKeyMan::FillPSBT and in LegacyScriptPubKeyMan::FillPSBT
< sipa>
SignatureData object is created, FillSignatureData is called on it, and then never used
< jeremyrubin>
I'm *pretty sure* it is a bug
< jeremyrubin>
It doesn't come up because I think if we're able to sign the input we erase the data and finalize it anyways
< jeremyrubin>
but it would be an issue in certain e.g. multisig cases I think
< jeremyrubin>
(finalize it, it = that input)
< jeremyrubin>
Well maybe it's not a bug; it just has no effect. But I think in theory we should be filling this information in from a different source and writing it back to the PSBT.
< jeremyrubin>
AFAICT there is no path for getting the witness_script (n.b. not scriptWitness) field filled in
< bitcoin-git>
[bitcoin] theStack opened pull request #20917: doc, rpc: add missing signet mentions in network name lists (master...2021-add_missing_signet_network_name) https://github.com/bitcoin/bitcoin/pull/20917
< jeremyrubin>
looks like passing sigdata to SignPSBTInput would be one step
< jeremyrubin>
ah it seems like that path does happen (sorry for the thinking out loud here)
< jeremyrubin>
There's another call which covers it
< jeremyrubin>
so we can probably just delete those temporaries? I'm just curious why they are there at all
< jeremyrubin>
I'm still unable to get it to fill out the witness_script fwiw
< jeremyrubin>
it produces a signature successfully
< jeremyrubin>
but
< jeremyrubin>
`bitcoin-cli walletprocesspsbt $(cat PSBT) false | jq '.psbt' | xargs printf > PSBT2` does not fill in the witness_script
< jeremyrubin>
even a fresh call to walletcreatefundedpsbt seems not to fill out this field. Maybe I'm missing something? But I thought this field is required to derive the scriptcode
< sipa>
what exactly is your wallet like?
< sipa>
it does look like a bug; i can't imagine that a wallet is able to sign, but if asked not to, doesn't fill in the witness_script
< sipa>
but it would be useful to be able to exactly reproduce the problem
< jeremyrubin>
yeah let me give you an exact repro script...