< luke-jr>
cfields: is there any way to get debug output to see why it's not using the cached builds?
< cfields>
jonasschnelli: thanks!
< cfields>
luke-jr: sure, but gitian makes it kinda tough. Can you drop to a shell in the builder?
< cfields>
luke-jr: first thing, try building one more time and seeing if it happens again
< cfields>
IIRC bionic's glibc was updated this/last week, so that might've triggered it
< luke-jr>
oh
< luke-jr>
pretty sure I've seen it multiple times in the last few days, but I haven't been watching closely enough to be sure
< cfields>
(depends does something like COMPILER_SEED=`$CC --version --verbose`
< luke-jr>
so I'll do a few more just in case
< cfields>
)
< cfields>
and if it changes, it invalidates all current builds
< cfields>
ok
< luke-jr>
(also, the cache files are replaced with the same filename)
< cfields>
oh, that's weird
< cfields>
the filename contains the hash of the determined build recipe. So that shouldn't be happening.
< cfields>
maybe some filesystem thing keeps them from sending correctly to the builder?
< luke-jr>
no sign of trouble there that I could tell
< luke-jr>
hmm
< luke-jr>
I have cache/bitcoin-linux-0.17/bitcoin-linux-0.17/, possibly from manual copy-from-target on failures
< luke-jr>
it's possible since the glibc bump, I've only had failures..
< cfields>
hmm?
< luke-jr>
cfields: when gitian fails, it doesn't copy cache changes back to the host; I'd intended to manually do that, but it's possible they ended up in the wrong place
< luke-jr>
combined with the glibc update you mentioned, that may explain why it's rebuilding every time
< cfields>
ah, makes sense
< cfields>
gitian builders: detached sigs for 0.17.0rc2 are up
< jonasschnelli>
MaxHastings_: does it mention that the current used bloomfilter false-positive-rates do not protect privacy? (See Jonas Nicks master thesis from 2015)
< jonasschnelli>
MaxHastings_: there are eventually non-protocol vulnerabilities (implementation issues)
< jonasschnelli>
as1nc: nice. You could mention pruning?
< jonasschnelli>
(and or electrum personal server)
< jonasschnelli>
But its OT in this channel... so use #bitcoin
< luke-jr>
cfields: can you sign PPC Windows builds too?
< luke-jr>
jk ;)
< cfields>
luke-jr: eh? does that exist?
< luke-jr>
cfields: dunno, hence jk
< cfields>
oh, phew
< luke-jr>
XD
< luke-jr>
(working on POWER8+ gitian binaries)
< luke-jr>
(for Linux)
< cfields>
because if they couldn't pull off small-endian arm, BE ppc wouldn't stand a chance :)
< luke-jr>
who said BE?
< luke-jr>
afaik even when Windows did support PPC, it was LE
< cfields>
yeah yeah, that's why I specified
< cfields>
wait, they actually did support PPC at one point?
< MaxHastings_>
jonasschnelli: Ah I see I did not know bloom filters were ineffective for keeping user privacy. Are there any other missing vulnerabilities on that page other than privacy concerns? I was told on Bitcoin slack that the SPV client could be tricked to change its consensus rules by malicious nodes.
< luke-jr>
cfields: NT supported a lot of archs
< sipa>
MaxHastings_: by definition, yes
< cfields>
huh. I used win embedded a few places, so I guess that makes sense. I never would've thought ppc though.
< sipa>
MaxHastings_: they rely on the majority of the hashrate
< luke-jr>
"Windows NT 3.51 added support for the PowerPC processor in 1995, specifically PReP-compliant systems such as the IBM Power Series desktops/laptops and Motorola PowerStack series; but despite meetings between Michael Spindler and Bill Gates, not on the Power Macintosh as the PReP compliant Power Macintosh project failed to ship."
< luke-jr>
not really sure what that means XD
< as1nc>
jonasschnelli, yes but i really want to incite people to txindex their chain so they can benefit the full spectrum of bitcoin capabilities. lightning for exemple require a txindexed chain right ?
< sipa>
as1nc: i don't think so (or that's temporary in any case)
< jonasschnelli>
Not sure. C-Lightning doesn't IMO
< as1nc>
lnd does i think
< sipa>
i'm sure that's temporary until better protocols to communicate exist
< sipa>
(like bip157)
< MaxHastings_>
sipa: Good to know.
< gmaxwell>
c-lightning doesn't.
< gmaxwell>
as1nc: txindex is non-viable long term.
< gmaxwell>
Anytime you think "X _requires_ txindex" you should think "X will eventually force people to depend on a centeralized service".
< molz>
lnd doesn't require txindex anymore
< as1nc>
ok interesting, I'm txindexing because I want to provide services for people. but the only case i find indexing worth it for personnal use is SPV wallets. does it makes sense ?
< as1nc>
i use electrum pointed to my node
< sipa>
as1nc: txindex is purely a local feature; it is not exposed over the network
< as1nc>
sipa, yes
< Dizzle>
as1nc: yep, it makes sense, for both personal use or providing centralized services. As for lnd, if you don't have a txindex, lnd basically falls back to rescanning blocks when it needs to know a tx's status, so a bit of a performance drag in some cases.
< gmaxwell>
how is scanning blocks going to be a performance drag? thats how a node works anyways.
< as1nc>
Dizzle: thx, so it make sense to txindex your chain for personnal use, even if, like gmaxwell said, its not viable long run
< gmaxwell>
Dizzle: I don't see any back and forth there, it just gets the block.
< gmaxwell>
txindex lowers performance a lot, so I'm still not seeing how you're saying the lnd performance is worth without it.
< luke-jr>
if only bech32 addresses were shorter, we could have called it belch32
< Dizzle>
gmaxwell: only saying the performance is worse without txindex for the cases when historical lookup of a tx's status is needed. I mean, that's what the index is useful for. The back and forth I'm referring to is getblockhash+getblock RPC calls for each height in turn, check for a match, repeat.
< gmaxwell>
Dizzle: and my point is that you're missing the forrest for the trees. fetching the list of transactions in a block is cheap. maintaining the txindex slows down block processing a lot.
< gmaxwell>
So I think what you're describing costs several milliseconds per block to save a millisecond on some blocks.
< luke-jr>
how does symbol-check not complain about aarch using symbols from glibc 2.17?
< luke-jr>
…oh, gitian disables it for aarch XD
< jnewbery>
gmaxell: "txindex lowers performance a lot" - is that still true after #13033 ?
< gmaxwell>
jnewbery: maybe it is no longer "a lot", but it will still be more costly than checking the txids in a block, since it does that and a lot more.
< gmaxwell>
jnewbery: I missed that PR, so thanks for pointing that otu.
< BlueMatt>
I mean I dunno what the IOPS cost of it is, but if you're running with huge dbcache it should be almost free modulo the fact that we fsync the block files all the time (which we should stop doing, really)
< sipa>
BlueMatt: once per hour
< sipa>
and only the ones written to afaik
< BlueMatt>
not for block files, no, every time we move to a new one
< BlueMatt>
last I checked
< BlueMatt>
yea, every time we move to a new file we FlushBlockFile
< BlueMatt>
which fsync's, afaics
< gmaxwell>
BlueMatt: your comment is frustrating me. Above we are having a discussion where the argument was made that a whole txindex should be enabled to replace simply checking each block for a transaction.
< BlueMatt>
ah, I missed context, sorry
< BlueMatt>
yea, obviously not
< gmaxwell>
perhaps txindex no longer slows down reindex by 20% or whatever. but yea...
< sipa>
performance isn't really the reason not to use it - it just doesn't scale well
< BlueMatt>
anyway, as a separate point, spinning-rust-ibd-with-huge-dbcache the hang after each "Leaving block file" entry is really annoying
< BlueMatt>
sipa: well, *also* performance :p
< BlueMatt>
sipa: wait, isnt extended-encoding the *only* way you can encode a 0-input 1-output transaction?
< BlueMatt>
reliably, at least
< BlueMatt>
short of psbt
< sipa>
BlueMatt: there is no reliable way to encode 0-input transactions
< BlueMatt>
sure, if you use extended-encoding
< sipa>
that's still potentially ambiguous
< BlueMatt>
the use-all-data heuristic should handle it fine
< BlueMatt>
and doesnt-need-to-be-backwards-compatible APIs (eg rust-bitcoin's deserializer) can *only* handle it that way
< sipa>
i'm confused
< sipa>
it is not legal to encode a tx without witnesses using extended encoding
< sipa>
so there should never be any ambiguity
< BlueMatt>
but every deserializer I've ever seen handles it fine, and its the only way to have no ambiguity in decoding 0-input txn
< BlueMatt>
so I think we should revise the spec to match implementations?
< BlueMatt>
or am I confusing myself
< sipa>
sigh, so make an exception to the only-extended-if-has-witness rule in case there are 0 inputs?
< sipa>
that's fine by me - i mostly care about only having a single encoding for actual transactions
< sipa>
using the raw tx format for partial transactions was always a hack at best
< BlueMatt>
yes, agreed, but the only reliably hack is always-use-extended, sadly :(
< BlueMatt>
but, ok, fair, only doing it for 0-inputs seems reasonable to me
< sipa>
we can also make the serializer do the same
< sipa>
always use extended encoding when 0 inputs
< BlueMatt>
would seem prudent to me :(
< sipa>
... or psbt
< sipa>
but everything at its time
< BlueMatt>
yea, well hopefully people migrate to psbt
< BlueMatt>
yea
< andytoshi>
achow101: how do you use PSBT when nobody knows the full set of inputs up front?
< andytoshi>
in particular, can you use PSBT to avoid the 0-input serialization ambiguity
< sipa>
right
< sipa>
for 0-input things people should just use a list of amounts/scriptpubkeys
< sipa>
(i've mentioned extended psbt to support 0 inputs just as a container format, though)
< andytoshi>
if we're going to propose an extension BIP to add p2c fields and industry/private reserved IDs, we should throw that in the pile
< sipa>
in any case, PSBT's serialization supports 0 inputs fine
< sipa>
(because it specifies using non-witness serialization for the tx, there is never any ambiguity)
< andytoshi>
oh right
< andytoshi>
ok i forgot that, never mind me, sorry
< sipa>
it's not really usable to use for the various PSBT roles just because there isn't anything to be done with it
< gmaxwell>
https://bitcoinperf.com/timeline/#/?exe=3,4,2,1&base=1+23&ben=reindex.522000.dbcache=2048.mem-usage&env=1&revs=50&equid=off&quarts=on&extr=on < I think this should be blocking 0.17.
< achow101>
BlueMatt: input txs are txs that already exist on the network .they cannot be 0-input
< gmaxwell>
BlueMatt: he's talking about prevout inputs.
< sipa>
BlueMatt: the inputs
< gmaxwell>
for checking fees.
< BlueMatt>
ah
< BlueMatt>
anyway, I'll stop complaining since I obviously didnt read the spec in detail
< sipa>
gmaxwell: in PSBT role terminology that would be someone who takes the output of a Creator, and then "creates" a new PSBT for a different transaction
< sipa>
gmaxwell: but in the workflow, all steps operate on the same "proposed transaction"
< sipa>
when there are 0 inputs, there isn't really a proposed transaction
< sipa>
though the same format could easily be used to support that
< achow101>
gmaxwell: psbt's workflows are currently only defined for the signing stage. it assumes that all inputs and outputs have been decided and negotiated in some previous step
< BlueMatt>
ah, so there are never any 0-input txn encoded in it at all?
< sipa>
BlueMatt: in the PSBT workflow, no
< achow101>
but you could still use psbt for negotiating the inputs. I'm not sure why you would though
< BlueMatt>
sipa: why didnt you say that in the first place? :)
< sipa>
but the format can be used for that, if you want
< achow101>
BlueMatt: no, there is not. but it technically still works and will be deserialized properly
< sipa>
18:55:17 < sipa> PSBT isn't designed for no-input cases
< sipa>
18:55:35 < sipa> it's to simplify the signing procedure
< sipa>
it's kind of annyong to think about PSBT-the-container-format (which supports 0 input fine, if you're willing to do the decode-knowing-there-is-no-witness thing) and PSBT-the-procedure-for-signing (which clearly doesn't) as separate
< BlueMatt>
well I mean tell me to shut up cause I havent read it in detail, but I think in a new container-format we should be encoding 0-input with extended serialization
< BlueMatt>
cause thats what we just said, above, too
< sipa>
i think that's insane.
< sipa>
PSBT implementations aren't even required to support segwit
< achow101>
BlueMatt: why? there are no witnesses to serialize ever
< sipa>
just add a flag to your deserializer
< achow101>
BlueMatt: a psbt has two types of txs: the unsigned tx (the one you want to sign) and the input txs (ones already on the network).
< achow101>
the unsigned tx, by definition, has no signatures. therefore it never has witnesses
< achow101>
thus it is completely unnecessary to serialize the unsigned tx with extended serialization as it can never have a witness
< BlueMatt>
sipa: we just discussed, above, encoding 0-input with extended serialization, since thats the *only* way to do it that is non-ambiguous
< BlueMatt>
and otherwise createrawtransaction+fundrawtransaction are...not ideal
< gmaxwell>
I'm just really confused by this discussion. I expected that PSBT was the fix for the createrawtransaction ambiguity case.
< gmaxwell>
and now you're saying that it isn't but it is?
< achow101>
BlueMatt: it is completely unambiguous because the unsigned tx is always non-witness
< gmaxwell>
so this discussion is pointless.
< sipa>
BlueMatt: this is the *first* time ever in my life i hear the suggestion to use extended serialization for 0 input
< luke-jr>
s/unsigned/no-input/
< sipa>
BlueMatt: it's a cool trick, i admit
< BlueMatt>
sipa: well its either that or we have to pass a flag into signrawtransaction (and any equivalent APIs in any bitcoin library anywhere)
< sipa>
but i had always lived under the assumption that our deserializer failed on that
< sipa>
it was just accidentally removed in a refactor
< BlueMatt>
of which there are many, any I generally dont trust most bitcoin libraries...for obvious reasons
< achow101>
BlueMatt: why are you signing a 0-input tx?
< achow101>
what is there to even sign?
< sipa>
achow101: not the point
< BlueMatt>
sipa: for p2p, it probably should, because, you know, 0-input is bogus on the p2p/consensus layer
< BlueMatt>
achow101: not signing, funding
< BlueMatt>
achow101: right now the suggested workflow of createrawtransaction, fundrawtransaction, signrawtransaction, sendrawtransaction is hit by the ambiguity bug
< gmaxwell>
achow101: See our rpc, "fundrawtransaction" --- that isn't "Fun! Draw Transaction." :P
< sipa>
BlueMatt: createraw and fundraw are merged into one in PSBT workflow
< BlueMatt>
its (mostly) fine right now, but my understanding was the psbt was addressing exactly that use-case but across multiple signers like multisig and the like
< luke-jr>
achow101: you don't know it's zero input yet
< gmaxwell>
the reason they exist seperately to begin with is that there are workflows where they need to be seperate. E.g. coinjoins.
< sipa>
FFS people
< BlueMatt>
sipa: fair, but you just mentioned that the existing container could be extended to split that out, and my point is, "is the container well-defined for the ambiguity bug"
< gmaxwell>
or partial coincontrols. ("spend _this_ coin plus, whatever is needed")
< sipa>
can you have commented on this 9 months ago?
< sipa>
BlueMatt: YES IT IS. NO WITNESSES IN THE UNSIGNED PSBT TRANSACTION
< sipa>
i'm tired of this discussion
< gmaxwell>
sipa: I read the spec and saw no evidence that 0 inputs wasn't a supported case!
< sipa>
the fact that you want to reuse your heuristic decoder for raw transactions in PSBT may mean you need to add a flag to disable that feature
< sipa>
in a post-PSBT world, nobody ever needs to implement such a heuristic again
< achow101>
"Type: Unsigned Transaction PSBT_GLOBAL_UNSIGNED_TX = 0x00 ... The transaction must be in the old serialization format (without witnesses)"
< sipa>
because actual transactions will always have witnesses (or use unambiguously old serialization for things without)
< sipa>
and PSBTs will never have witnesses
< gmaxwell>
it's surprising to me that you and achow didn't except PSBT to be a superset of the applications of the existing raw transactions.
< gmaxwell>
achow101: great, and the old tx format supports 0 inputs.
< achow101>
gmaxwell: exactly
< sipa>
gmaxwell: not... really
< achow101>
gmaxwell: but it is _always_ the old serialization. no ambiguity there
< sipa>
this discussion only happened because someone only has a heuristic tx decoder in their software, and finds it annoying to add a way to disable that heuristic
< gmaxwell>
achow101: sure, I agree. Sipa was asking why wasn't this commented on 9 months ago. And my reply is because the shortfall (that 0 input PSBT aren't a supported usecase) isn't something I could extract from the spec.
< luke-jr>
achow101: the ambiguity is because you don't know if it's 0-input or 1+ input segwit until you parse it correctly
< sipa>
luke-jr: you're missing context
< gmaxwell>
luke-jr: it's not ambigius in PSBT.
< luke-jr>
ok
< gmaxwell>
In PSBT the internal tx never has witnesses.
< achow101>
gmaxwell: the format supports 0-inputs just fine. I think there's even a test for it. the processes that we also defined in the bip do not.
< sipa>
gmaxwell: i think i always thought of PSBT as a better unambiguous serialization... but the BIP describes both a serialization and the procedures around it
< achow101>
but the processes don't matter that much. you can use the psbt format for whatever steps that occur before signing and then use the processes defined in the bip
< sipa>
and those procedures don't really map well to the part where you're still deciding what exactly to sign
< gmaxwell>
I never read that stuff, it's not normative (doesn't change the meaning of the datastructures), and expirence from other BIPs suggests tha people ignore them. (see, for example BIP32)
< gmaxwell>
(or at least don't read it carefully)
< sipa>
gmaxwell: well, then you were right
< sipa>
that part of PSBT supports 0 inputs fine :)
< gmaxwell>
sipa: thanks! confusion resolved!!
< jnewbery>
sipa: performance isn't really the reason not to use it - [txindex] just doesn't scale well" - scale well in what regard? Space requirements should be linear with blockchain size for a full archival node, right?
< sipa>
jnewbery: i don't think it is realistic that everyone needs to store the whole chain
< sipa>
or have fast access to it
< jnewbery>
I agree! But for those that do, txindex is just a constant factor increase in space requirements
< sipa>
sure
< jnewbery>
ie txindex scales as well as archival nodes scale
< sipa>
but building solutions that rely on having a fully-indexed blockchain available isn't scalable
< sipa>
you don't need many archival nodes
< sipa>
furthermore, nothing really _needs_ access to the full chain if designed well, in my opinion
< sipa>
except for debugging purposes
< jnewbery>
a block explorer?
< sipa>
for example
< sipa>
(i count that as debug purposes)
< sipa>
not something to rely upon for production
< jnewbery>
I agree that products shouldn't *rely* on block explorers, but there's demand for them now and I can't imagine there'd ever not be demand for them
< sipa>
sadly enough :(
< sipa>
however, in general i think we should discourage building on things like txindex
< jamesob>
and in any case it seems like there'll always be use-cases which require rescan; ie when you become interested in an output you weren't keeping tabs on before
< sipa>
as i fear we may end up with a less scalable ecosystem if everyone assumes it's easy to just look up whatever transaction
< sipa>
and of course there are use cases - it wouldn't be there otherwise
< sipa>
jamesob: i think that's either sign of a badly designed system, or an emergency (for which i think it's fine to need some index, you could also pay someone to look things up for you)
< jamesob>
yeah, maybe so
< roasbeef>
lnd doesn't require the txindex, it'll detect if the full node has it on or not, and fall back to just fetching blocks it needs manually, eventually if the gcs filters are exposed on RPC it could also fetch those before going for the full block it needs
< jnewbery>
It's fair to say that 'assuming that it's easy to just look up whatever transaction' isn't a scalable architecture to build a product on, but I don't think the txindex in itself is unscalable now that we #13033
< roasbeef>
jnewbery: things could also be split up into distinct database, or even really just like memory mapped direct hash tables, shoving everything in leveldb makes other routine stuff slower as you end up with thousands of files that all need to be memory mapped and also to watch out for fd limits, then on top of that you end up with a handfull of disk seeks due to the file and sstable structure to read something like the offset of a txid in a block
< jamesob>
roasbeef: jimpo's txindex change already moves indices into separate leveldbs
< roasbeef>
Dizzle: we now continually also update a "height hint" so really a checkpoint of the state of that outpoint/txid of its best known state, so like if we know it isn't conf'd at height 100, but it was broadcast at height 10 we know now to start at height 10
< sipa>
roasbeef: BIP157 will improve things further, i assume?
< Dizzle>
roasbeef: nice
< jnewbery>
ok, yes - that is fair, although in practice I'm not sure how much performance impact there is. If it's truly a performance bottleneck, then I assume #13243 gives us the infrastructure to store txindex somewhere else
< roasbeef>
sipa: yeh, either if it's going over p2p network, or just even getting the filters over rpc from the full node, as it can fetch the filters in batch, and also cache filters on its end
< jamesob>
jnewbery: indexes are already isolated from other dbs in /datadir/indexes/[index name]
< roasbeef>
also eventually the stuff about creating larger like hierarchical multi-block filters can help as well when the node has been offline for a long-ish time
< gmaxwell>
Under a no-address reuse assumption (lol), hierarchical filters are never a win.
< gmaxwell>
13:16:57 < jnewbery> ie txindex scales as well as archival nodes scale
< gmaxwell>
I don't expect that archive nodes will be a thing in 10 years, outside of academic curiousity and what not.
< gmaxwell>
No more than traces of all announced transactions from all peers in the network are a thing now.
< gmaxwell>
I expect that new nodes will sync up using a months old UTXO state thats vallidated according to the assumevalid model, then catch up.
< jnewbery>
Seems possible, but for the next 10 years I expect at least some people will want to run archival nodes!
< gmaxwell>
oh absolutely, and we should do what we can to make that work well too.
< sipa>
i hope they do.
< sipa>
but i hope no services realpy need them
< gmaxwell>
I've pushed a lot on optimizations related to e.g. reindex/ibd in the last year specifically because once we're syncing from AV-UTXO there will be much less motivation to improve these things.
< jnewbery>
to be clear - I'm not arguing that we should *encourage* the use of txindex or needing a full blockchain. I just wanted understand what was meant by "txindex is unscalable".
< sipa>
right; i think it's more "an ecosystem relying on txindex doesn't scale well"
< jnewbery>
yes, makes sense
< sipa>
txindex as an algorithmic thing itself scales as well as it could i think
< gmaxwell>
Well I'm sure it's lacking a ton of possible optimizations.
< sipa>
sure, but no dramatic complexity changes
< gmaxwell>
no, just constant factors.
< gmaxwell>
it's not the txindex structure itself isn't scable, it's that operations over the chain history aren't scalable.
< jnewbery>
sipa, has your opinion changed at all since this: I both hate and love this patch. I hate making it easy to build infrastructure that relies on a fully-indexed blockchain (which shouldn't be necessary), as it may remove the incentive to build more scalable solutions. On the other hand, in cases where the alternative is relying on a trusted third party, this approach is certainly preferable, and
< jnewbery>
would allow an RPC-based blockexplorer, for example.
< gmaxwell>
RPC based explorer also needs a reversed spentness index.
< gmaxwell>
jnewbery: it's an interesting question of if people would actually use it if it was provided. If they would I think it would be good to have. ... at least for the next 10 years, _some_ users would stop bleeding privacy wise.
< sipa>
jnewbery: i have the same love-hate relation with it :)
< instagibbs>
we(not me clearly, someone else) kind of broke the call. Probably is a way around it(walletupdatepsbt?)
< instagibbs>
walletprocesspsbt
< gmaxwell>
Well I think unless it turns out that the memory usage increase is fictional, we'll have another RC anyways.
< instagibbs>
phew
< luke-jr>
woo finally got powerpc64 builds out of gitian working
< luke-jr>
just in time for bluematt to close his PR :x
< luke-jr>
is it preferable to de-bundle libpng from depends/qt, or patch the libpng bundled with depends/qt?
< luke-jr>
cfields: ^
< luke-jr>
probably debundling, since Qt's copy is missing the files needed :/
< * luke-jr>
ponders using virtfs for gitian cache
< gmaxwell>
jonasschnelli: so this protocol's use of a seperate chacha20 for the sizes is effectively halving the performance of our cypher.
< gmaxwell>
jonasschnelli: the use of a message number as the nonce (instead of using a continious stream) will also decrease our performance for AVX by maybe 20%.
< sipa>
gmaxwell: that's a strange gratuitous performance reduction, indeed
< sipa>
i wonder why openssh did that
< gmaxwell>
maybe because they thought they would read ahead lengths and thus schedule the other runs more efficiently?