< jimmysong>
storing that data in the db for a full node so it can be given to a peer that requests it
< sipa>
a bloomfilter (or gcd filter) can't store any data, it's just a probabilistic structure to let you query for set elements
< sipa>
a full node has no need for that data
< jimmysong>
a full node doesn't but it'd be useful for a light client wanting to verify a cfilter hash
< sipa>
it is store on disk though, in the undo block data in bitcoin core, for as long as the block itself is stored
< sipa>
ah, for verification it should just be committed to
< jimmysong>
a coinbase commitment is better, but is that something that's got a proposal?
< sipa>
it's pretty trivial to do, but mucb easier to get agreement on once bip157 itself is deployed and used
< sipa>
i expect
< jimmysong>
makes sense
< jimmysong>
i'm asking because i'm trying to write a wallet that uses neutrino
< jimmysong>
the annoying part is the verification
< jimmysong>
but getting the filter, checking against my script pubkeys, etc is great
< jimmysong>
much better for privacy
< sipa>
the node giving you prevour scripts along with the filter doesn't let you verify anything
< sipa>
they can forge anything
< jimmysong>
it lets you verify that the filter bytes are what they said
< jimmysong>
and you can check that the scripts for the inputs verify
< sipa>
no, because the prevout scripts can be forged
< sipa>
there is nothing committing to them
< jimmysong>
wait, if you have a signature, you can make a pubkey that works with it?
< jimmysong>
err, rather, a pubkey, you can make a hash preimage?
< jimmysong>
the scriptsig/witness for p2pkh/p2wpkh is pubkey, sig
< jimmysong>
so you'd have to forge a preimage of the pubkey
< sipa>
oh, you mean full script validation?
< jimmysong>
yes
< sipa>
that shouldn't be something a light client cares about
< jimmysong>
if it helps verify the cfilter hash, why not?
< jimmysong>
commitment is obviously preferable
< jimmysong>
but if we don't have that?
< roasbeef>
jimmysong: yeh there's a half proposal on the ML to add that, the prior versions had the outpoint so they were fully verifiable, as is now you can verify half of it
< sipa>
what is your attack model?
< roasbeef>
differntiate honest peers from dishonest peers
< sipa>
you don't want to get the prevout script for every block, right?
< roasbeef>
it's not full script validation, it's verifying the filter is correct
< sipa>
sure, but you'd need a full script interpreter for doing so
< roasbeef>
yeh you'd fetch the block with the prior scripts and value why not while you're at it, w/ value you can verify fees to a degree as well
< roasbeef>
no need to verify the script, just to verify a filter actually fully matchs a given block
< jimmysong>
i was thinking full script interpreter
< jimmysong>
which isn't trivial
< sipa>
roasbeef: sure, that makes sense; but jimmysong was suggesring script validation, as without it, the prevout data can be trivially forged
< roasbeef>
txscript.VM ;)
< sipa>
jimmysong: but for which blocks would you get this prevout data?
< jimmysong>
the blocks you download when you search for your utxos
< roasbeef>
blocks for which you get conflicting advertisements
< jimmysong>
that too
< sipa>
jimmysong: a peer can still lie and give you a filter with no matches at all
< roasbeef>
you can fetch the block and verify the outputs are properly included, but not fully inputs
< sipa>
yes, for confloct detection it makes sense, and you need no script validation etc
< roasbeef>
main purpose if conflict detection
< roasbeef>
is*
< roasbeef>
harding: the prev outs are in the undo blocks
< roasbeef>
this was a concern during the switch over (that it would be slower to construct our possibly impossible), but the data is still around
< jimmysong>
what's the objection to writing a script interpreter?
< jimmysong>
is it because of all the edge cases?
< roasbeef>
many already exist
< jimmysong>
i've written a partial one
< jimmysong>
i still don't get op_codeseparator
< roasbeef>
code sep should die, ppl should stop trying to save it lol
< jimmysong>
btw, roasbeef, those test vectors for neutrino broke my script interpreter =)
< sipa>
jimmysong: in my view script interpreting is so easy to get wrong, it should only be done by full nodes
< jimmysong>
it is easy to get wrong, but isn't there value to at least interpreting some subset?
< jimmysong>
p2wsh/p2sh stuff can get pretty complicated
< sipa>
i don't think so
< sipa>
a wallet can recognize its own outputs, bit that doesn't need interpretatiom
< roasbeef>
you can't even verify the outputs fully exist, so script verification is w/e, main reason for getting the prevouts is filter verification and fee awareness
< sipa>
for debugging purposes it can be useful of course, but imo that's it
< jimmysong>
so wallets should stick to standard scripts, or whatever has been analyzed beforehand
< sipa>
well they construct them themselves
< sipa>
so you don't need an interpreter to find its semantics
< jimmysong>
right, the ones analyzed beforehand by the programmer
< sipa>
and you won't trigger any edge cases unless you go out of your way to trigger them
< jimmysong>
in any case, if the outpoint data is in the undo blocks, that can be made available?
< roasbeef>
yep, just add a new inv type, and message format
< jimmysong>
sure, you need the full tx's to verify against the hashes, which as harding pointed out can get pathologically large in edge cases
< jimmysong>
similar to jj's attack from breaking bitcoin last year
< sipa>
jimmysong: needing the full txn is unrealistic
< sipa>
that data isn't available in general, and expensive to index
< roasbeef>
why do you care about the hashes? get the script+value, verify the sig+fee, reconstruct filter, dunzo
< roasbeef>
ofc if the witness commitment was to a merkalized version of the transaction, you could fetch that proof as well, but mo bytes there, what we have available atm is good enough to achieve the goal of cross checking
< sipa>
in general we should aim to only expose data that is verifiable or can be made verifiable
< sipa>
and adding comitments to undo data seems overkill, when all that's needed is a commitmemt to the filters
< jimmysong>
yes, that's the preferred solution
< roasbeef>
yeh not undo, just filters eventually, mean as in if the wtxid was a merkle tree with leaves of the inputs/outputs etc
< roasbeef>
would also let you have a more compact proof of spentnesss, especially if things are also mega super coin-joiny in the future
< jimmysong>
wow, good point
< sipa>
i think all these adhoc methods of verifying filters break down if you look at them in an adverserial setting... you can use heuristics and download from multiple peers, and detect conflicts, and spot check here and there... but those approaches all add complexity and bandwidth proportionl to the level of guarantee they give
< roasbeef>
yep those other appraoches are just plain easier to mess up, i prefer dead simple methods
< sipa>
the only foolproof, cheap, ... way is a filter comitment in blocks
< jimmysong>
yep, which costs something for a miner, but not very much. but it is a soft fork
< jimmysong>
any idea how that would be received?
< gmaxwell>
the bigger limit there is just that there needs to be a higher level of confidence in the design, since removing the requirement is a hardfork. Fortunately, as it is now has actually had a fair level of refinement.
< roasbeef>
jimmysong: filter commit in op return, message to fetch header along w/ path for coinbase transaction? i think the filter chain would prob still have some use in that case as well
< gmaxwell>
I've suggested in the past potentially doing a kind of rolling softfork, where the commitment is required to be valid only if its provided and only until some height. And so long as people keep using it and it isn't replaced with something better, we just keep soft-forking in updated heights.. but if ever it gets replaced, it can just be allowed to expire.
< jimmysong>
that would be a nice escape clause
< roasbeef>
oooOOo, i guess the witness commitment is kinda like that for blocks w/o any witness data
< sipa>
jimmysong: i think it's necessary; without commitment to the filters you can basically only use it locally or with a trusted full node
< sipa>
which is pretty useful on itself, but nearly the same
< sipa>
but not nearly
< gmaxwell>
I'm very glad, e.g. that nothing about BIP37 ended up softforked in... that protocol turned out to be a lemon in a number of ways, but it took a couple years of use to realize that.
< roasbeef>
well you can still use it in the open net, you have the same "one honest peer" assumption going on, the scripts just help you to distinguish between zero and 1 honest peer
< gmaxwell>
roasbeef: even 1 honest peer requires you to have some kind of complex resolution to deal with disagreement, also-- because the p2p network is trivally sybable 'one honest peer' probably doesn't mean much unless you're doing something like manually configuring a trusted one.
< gmaxwell>
One should also consider the effect of incentiving varrious kinds of trouble making. (like generally we've found that when we added vulnerablities like BIP37 attackers emerged that didn't exist previously, and then made more trouble even for people that didn't care about using BIP37)
< roasbeef>
gmaxwell: you'd just fetch the blocks+scripts and reconstruct the filter, the commitment to the filters (the filter chain) helps you notice if something is funky at a glanec
< roasbeef>
but then again, for smaller values you're prob not super concerned about this stuff
< gmaxwell>
roasbeef: matching the spent inputs too requires fetching the inputs, which as sipa pointed out above is intractable in the worst case (and also astronishingly expensive even in the normal case)
< roasbeef>
yeh the assumption is a new inv type to allow you to fetch the the input scripts, i guess i miss this worst case scenario?
< roasbeef>
fetch with the block*
< gmaxwell>
keeping also in mind that this stuff is saving you 30kbit/sec of bandwidth over downloading the whole blocks... in the ongoing case.
< gmaxwell>
roasbeef: we have no way to provide that, and even if we did it couldn't be validated without fetching potentially gigabytes of additional data (you actually have to fetch the whole transactions).
< roasbeef>
ahh ok yeh i was missing the outpoints in the equation
< sipa>
roasbeef: what if you receive two different filters, and request block/prevout data for both, and they are both correct? (and it turns out you receiced one true prevout data and one false prevout data, but with correct filters for that data)
< roasbeef>
well can still verify half of it today ;)
< gmaxwell>
roasbeef: yes, half indeed. and really the more useful half.
< gmaxwell>
but at the expense of fetching the whole block.
< sipa>
that's still detectable, but you're rediced to a majoroty vote kind of model, and at a possibly very high bandwidh cost
< gmaxwell>
which also means that it really only takes one clown with an aws instance to cause a lot of users to fetch the blocks anyways. I mean, perhaps not totally useless. But as I mentioned, vulnerablties seem to attract attackers.
< gmaxwell>
so then you're left with the work of implementing the validation, testing it, dealing with vulnerablities in it... and some clown spins up a bunch of sybils and everyone is downloading the entire blocks anyways. And-- the sybils themselves act as a lasting nussance. I don't mean to argue against the non-commited usage, but considering the all in effects it's not the obvious big win that it
< gmaxwell>
might seem at a glance.
< gmaxwell>
and the historical rate of protocol additions introducing vulnerablties (either in implementation or design) is really high...
< roasbeef>
it's a win for full nodes at least, serving the filters is much less intensive (and also stateless) compared to serving bip37, you also can't trigger worst case matching behavior over the entire chain
< gmaxwell>
roasbeef: quite a few nodes just disable BIP37 completely (which seemed to stop the BIP37 based attacks)
< roasbeef>
yeh i ended up doing that on my testnet nodes, seemed somsone was practicing their attacks on testnet lol
< gmaxwell>
(I'm not disagreeing with your point though)
< gmaxwell>
roasbeef: they were doing it on mainnet for a while. Though they seemed to give up after even a small set of their targets started disabling them. To the extent that they might have been targeting miners to cause block orphaning, that makes sense, but otherwise it's not really clear why it stopped.
< gmaxwell>
Perhaps because they realized if they kept it up BIP37 was just going to end up removed and then they'd lose their toy. Who knows.
< fanquake>
Clearly not bothering to read my comment, or from what I can tell, anything related to actually contributing to the project.
< gmaxwell>
I'd recommend just closing and locking the PR. They aren't following the guidelines, and there is a good chance that its just trolling.
< gmaxwell>
no different than any other driveby pr
< gmaxwell>
and especially due to the principle that the project isn't a place that we'll tolerate other people turning into their performance art or battleground.
< gwillen>
fanquake: if you are able, it seems like preemptively locking might be a good idea.
< fanquake>
I was going to leave it for a reply, but have done it now.
< gwillen>
the quality of his previous comment doesn't make me sanguine
< gwillen>
also, the account that hit 'approve' looks like a sockpuppet
< fanquake>
wumpus are you around tonight?
< bitcoin-git>
[bitcoin] hebasto opened pull request #15038: docs: Get more info about GUI-related issue on Linux (master...20181226-issue-template-gui-linux) https://github.com/bitcoin/bitcoin/pull/15038
< wumpus>
fanquake: maybe a bit
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #15039: wallet: Avoid leaking nLockTime fingerprint when anti-fee-sniping (master...Mf1812-walletLocktimeFingerprint) https://github.com/bitcoin/bitcoin/pull/15039
< fanquake>
wumpus np, bunch of PRs mergable, but can deal with em later
< wumpus>
PSA: there's no meeting today (there was confusion about this last week)
< hebasto>
wumpus: today is Wednesday. Do you mean tomorrow?
< wumpus>
oh sorry yes I mean tomorrow
< fanquake>
only Wednesday for another 41 minutes anyway
< sipa>
wumpus: no, you're right, no meeting today!
< andytoshi>
are there any guarantees about the order of `getrawmempool` output? in particular do ancestors always precede descendants?
< instagibbs>
from my cursory reading they are indexed in 4 different ways, none of which are related to ancestors/decendants
< andytoshi>
kk thanks, i won't rely on that then
< instagibbs>
salted txid, feerate(including desc), entry time, and sorted feerate(including ancestor
< andytoshi>
i mean, somehow when creating blocks they have to wind up in ancestor order ... is there an explicit sorting step then?
< instagibbs>
they grab "packages" as they sort via package feerate
< instagibbs>
there are internal links between the entries, just not exposed here afaik
< andytoshi>
ah, yep, that makes sense
< andytoshi>
my goal here is to recreate the packages from the output of getrawmempool
< sipa>
andytoshi: i'm pretty sure they're sorted by increasing total number of (recursive) unconfirmed dependencies, and then by feerate
< sipa>
and then by txid as tiebreaker or so
< sipa>
which guarantees that dependencies always come before the dependendees (?)
< sipa>
that code is not shared with the block creation code, btw
< instagibbs>
andytoshi, also try getmempoolentry which comes with more details?
< andytoshi>
instagibbs: oh, yeah (or `getrawmempool true` which gives me the same details). will take a look at that to see if it's useful. i suspect not, i think i need to manually recreate a lot of this data in my code because i need a bunch of extra details, like the set of yet-ununspent inputs/outputs for the whole package
< andytoshi>
sipa: cool, thanks. but i guess that's an implementation detail and if i were to write production software that depended on it, you'd be annoyed :)
< andytoshi>
i wish there was some way i could signal core that i don't want certain outputs to be 0conf-spendable (or if they are, that i don't want cpfp rules to be applied)
< gmaxwell>
andytoshi: encountering the RBF pinning problem?
< andytoshi>
the issue is that if my software is making a package A of transactions and doing all sorts of CPFP logic to make that sensible, and meanwhile some customer of mine is creating a package B with a massive low-fee sweep or whatever
< andytoshi>
and then that customer creates a tx spending an output of A alongside the output of B...
< andytoshi>
...those packages become merged and suddenly my logic has been blindsided
< gmaxwell>
that isn't how the implementation works.
< gmaxwell>
I think Russell is like... reasoning from the feature's name.
< andytoshi>
well, the code is not super straightforward to someone uninvolved with it. the mempool logic related to the descendant limit looks kinda like it would do stuff like what i described
< gmaxwell>
The descndant limit stuff can do things along those lines, but requires actually hitting the descendant limit.
< gmaxwell>
If the limits are getting hit by ordnary usage we should look into fixing that. They were set so they were never hit when established (except for some obviously dumb floody crap), and only exist to prevent a bad computational blowup in the tracking code.
< instagibbs>
one thing to note is that a single ~100kvB sweep can hose the descendant size limit pretty much
< gmaxwell>
so fix it?
< gmaxwell>
IIRC the reason for the size limits in the tracking is just so it doesn't falsely credit parents for feerate coming from transactions that are never going to fit in the same block.
< andytoshi>
maybe i'm confused about cpfp. my understanding is that if i make a tx with outputs controlled by other people, those other people are able to grief me and undermine my ability to use cpfp
< andytoshi>
by extending the package such that i'd be hitting limits
< gmaxwell>
yes, though also at the expense of delaying their own transaction confirmation.
< gmaxwell>
I don't believe we've ever seen cpfp 'griefing' reported.
< gmaxwell>
RBF pinning for sure, because a common usage pattern immediately causes it.
< gmaxwell>
The limits exist only because there are computational overheads in the tracking, e.g. when removing a transaction its ancestors and descendants need to be walked to update their tracking.
< andytoshi>
sorry for the dumb questions, but can you clarify - if i'm making cpfp packages, and one of my customers 0conf spends one of the outputs i create, will their transaction pull my effective feerate toward the feerate of that tx? if so, then i need logic to reason about that, and by nature that logic needs to know about the limits (even though in practice i don't expect anyone to pull me close to
< andytoshi>
them)
< andytoshi>
or is it safe if i just make my own transactions that chain off each others' change outputs, and ignore everything else?
< andytoshi>
maybe i should just pester sipa in person in a couple of weeks :)
< andytoshi>
and i will try to write down what i'm learning as i do this
< gmaxwell>
No, it will not.
< gmaxwell>
The parents effective feerate is the highest feerate you can construct with it.
< andytoshi>
ok, i think i've got it
< gmaxwell>
They can, if they spam out to the limits, prevent new descendants from being taken.
< gmaxwell>
But thats only in the case that the tracking limits are hit.
< andytoshi>
so is this a rough high-level view of cpfp in core?: (a) "packages" only exist during miners' transaction selection; in the case that a transaction might be in multiple packages, they're computed greedily to maximize feerate; (b) but when accepting to the mempool, Core checks whether a transaction might cause a limit-violating package to exist, and if it would, the tx is rejected
< gmaxwell>
close enough; the limits aren't really limits on 'packages', they're more limits on the tracking datastructures used to create packages.
< andytoshi>
yep, that's what i meant
< bitcoin-git>
[bitcoin] hebasto opened pull request #15040: qt: Add workaround for QProgressDialog bug on macOS (master...20181226-fix-macos-qprogressdialog) https://github.com/bitcoin/bitcoin/pull/15040
< jnewbery>
andytoshi: I think that's mostly right. In (a), the miner is ordering by ancestor feerate (see BlockAssembler::addPackageTxs in miner.cpp). In (b), all of the {(ancestor|descedant) (count|size)} are taken into account (see CTxMemPool::CalculateMemPoolAncestors() in txmempool.cpp)
< gmaxwell>
jnewbery: your last statement sounds confusing, and plays into roconnor's misunderstanding.
< gmaxwell>
Basically what roconnor was thinking was that if there is an unconfirmed txn and then I add a gigantic low feerate child to it, I lower the feerate of the txn because the "package" has a lower feerate.. And that is not how it works, because of the max operation in the combining.
< jnewbery>
(b) is not looking at feerate. Just tx count and size