< devmob>
anyone here I can talk to about bitcoin's p2p layer ?
< devmob>
dns seeds, peer discovery and so on..
< grubles>
ask your question :P
< wumpus>
devmob: yes, sure, ask ahead
< wumpus>
depending on how technical the question is, #bitcoin might be a better place to ask if it's about high-level concepts and not implementation
< jl2012>
is it possible to return the index of a tx in a block, using getrawtransaction with -txindex?
< gmaxwell>
jl2012: the interface doesn't do that, but the data is there and it could.
< gmaxwell>
jl2012: the txindex tells the node what block the tx is in, it reads the whole block to give the result.
< jl2012>
gmaxwell: should I look at GetTransaction() ?
< sipa>
yes
< sipa>
the index also has the position, i think
< gmaxwell>
oh it does? hm.
< sipa>
ah no, it has the byte offset w.r.t. the block position
< jl2012>
CDiskTxPos postx is the byte offset?
< sipa>
right
< wumpus>
yes also checked, it only has the byte index
< jl2012>
is it possible to turn this into the tx index?
< sipa>
?
< sipa>
what do you need it for?
< jl2012>
I have some txids and need to know their order in the chain
< gmaxwell>
well if you only need to know their order, ...
< wumpus>
only if you'd read the entire block from disk, and parse that, you can figure out what the transaction index in the block is
< wumpus>
then byte index will work just as well
< gmaxwell>
well block number || byte offset in the block gives you the ordering of transactions.
< wumpus>
yes
< jl2012>
yes, that's enough for my purpose. Do you think it's a good idea to return the byte order in getrawtransaction?
< sipa>
i think getrawtransaction should just return the raw transaction
< jl2012>
sipa....yeah....but we could have a different name for that....
< gmaxwell>
jl2012: we probably shouldn't return anything that is implementation dependant unless we have a good reason.
< jl2012>
gmaxwell: yes, that's why I wonder if I could get the tx index
< gmaxwell>
E.g. we shouldn't add things to our API that would get in the way of changing to a better txindex or block storage design, unless there is a clear reason.
< wumpus>
at least if you do it, return the byte index relative to the block, not to the file start
< wumpus>
otherwise it'll be used to directly seek into the block files, it was tried before to add block filename/offsets to getblock and rejected for the same reason
< gmaxwell>
even that, for example we know that it's possible to store txn a lot more compactly than we do, and save maybe 25% of block storage. If we switch to storing blocks that way, those numbers either all change around or to keep them the same we'll have to store extra data or reseralize the block to return a result. :)
< wumpus>
gmaxwell: what if we add a warning to only use it as opaque ordering token
< wumpus>
don't call it byte offset
< sipa>
it's also not available for mempool txn
< wumpus>
unless that would reorder the transactions as well of course
< wumpus>
for the mempool it certainly wouldn't make sense, what would offset into the mempool even mean
< gmaxwell>
wumpus: that would be fine. Nah, using a more efficient serialization wouldn't reorder anything.
< gmaxwell>
wumpus: (uint64)txn* :P
< jl2012>
gmaxwell: and the order itself is consensus-critical so you need to store it somehow
< sipa>
jl2012: consensus enforcement doesn't even require storing the blocks at all :)
< sipa>
let alone an index
< sipa>
and more practically, it would not be outrageous i think to store the entire block by gzipping it in its entirety, at which point there is no more per-tx access possible
< sipa>
(i'm not actually suggesting this; we can do much better than that without giving up the ability to access txn individually)
< wumpus>
you'd still need an the offset after unzipping
< wumpus>
to retrieve it
< gmaxwell>
jl2012: to serve up old blocks we need to be able to recover the order, that doesn't necessarily mean that it's easily accessible at random. :)
< sipa>
wumpus: well, only if we care about retaining the ability to find individual txn
< wumpus>
I think some compression algorithms do allow for seeking by storing more decompressor state
< sipa>
wumpus: which is really just to speed up getrawtransaction
< wumpus>
sure...
< gmaxwell>
wumpus: yes, but in doing so more or less eliminates the gains from compressing things as a unit.
< sipa>
sorry, too many hypotheticals here :)
< wumpus>
anyhow I don't think forever API compatibility is so important for this, if we can't provide it anymore ,drop it then
< jl2012>
sipa: generating SPV proof is also a consensus-critical process
< gmaxwell>
jl2012: unclear what you mean by that.
< sipa>
jl2012: no?
< sipa>
validating of new blocks doesn't need SPV proofs
< wumpus>
gmaxwell: right, and the problem is that the decompressor state can be quite big compared to just a pointer, maybe bigger than the tx itself :-)
< gmaxwell>
The bitcoin protocol doesn't generate SPV proofs for arbritary transactions, it certantly isn't consensus critical to do so! :)
< gmaxwell>
in any case, other than potentially bulk compressing a whole block at a time, I don't see why a "this isn't the size but it is ordered" value would get invalidated.
< wumpus>
right, potentially you could still store it if you want to keep it available on the RPC API, even if you don't *need* it for the index
< gmaxwell>
right, and that would be totally worth doing if there were a known usecase for the interface... :P
< wumpus>
yes...
< gmaxwell>
though at that point, it would probably be best to just store a plain total order number.
< jl2012>
gmaxwell, sipa: maybe I'm using a wrong term.....anyway, I mean the index is critical for a full node to properly function (e.g. serving SPV proof, helping other nodes for IBD)
< sipa>
no it isn't
< sipa>
it's optional
< sipa>
the index isn't even used when helping other nodes IBD
< sipa>
(which on itself is optional)
< jl2012>
sipa: when you transmit a block, you also transmit the index (implicitly)
< sipa>
ah
< sipa>
by index you mean transaction position
< sipa>
not "the txindex database"
< sipa>
sorry!
< gmaxwell>
jl2012: bitcoin core doesn't even serve SPV proofs for arbritary transactions today!
< gmaxwell>
jl2012: and yes, to serve out a block you must be able to recover the order, but that doesn't mean that its stored in a way that allows efficient random access to it.
< gmaxwell>
E.g. we could store groups of blocks 100 at a time, gzipped. We could still serve up access for syncing peers. But order wouldn't be efficiently accessible for a txindex.
< gmaxwell>
(and any ability to serve a spv proof at all was only added in v0.8 and can be disabled with a commandline flag)
< gmaxwell>
also a full node doesn't even need to have historic blocks at all.
< gmaxwell>
Full node means that it autonomously enforces the consensus rules for itself without trusting third parties to validate, thats it.
< gmaxwell>
in any case, all I'm trying to say is just because the data is currently there isn't a reason to expose it in an API. Doing so is a pretty severe api anti-pattern. The purpose of an interface is to encapsulate complexity. Any time we expose an internal we make it more costly to change in the future. There are known things people are working on that if they were adopted would change the positio
< gmaxwell>
n values, though they would at least keep them ordered.
< jl2012>
gmaxwell: I understand your point........could we have some kind of "expert API" that does not guarantee to work in the future?
< sipa>
what do you need it for? perhaps there are other ways
< jl2012>
sipa: I have some random ordered txids, and want to reconstruct the ledger.
< jl2012>
the ledger of certain addresses
< jl2012>
a dumb way is to call 'getblock' to learn the order.....
< jl2012>
efficiency is not a big issue for me so that's probably good enough
< jl2012>
but there is another problem: I just find that getrawtransaction doesn't return block height!
< jl2012>
only block time and #of confimations.....so I need getblockchaininfo to tell me the current height....
< roasbeef>
jl2012: verbose should, well returns confirmations which you can use to compute the blockheight the tx was incluided in
< jl2012>
roasbeef: yes, just need to do one more step.....
< jonasschnelli>
jl2012: can't you keep a headers chain in your application and look up the blockhash there?
< jl2012>
jonasschnelli: getrawtransaction provides only "number of confirmation" and "blocktime", but blocktime couldn't be a reliable block index because it is not strictly ordered and might repeat
< jl2012>
jonasschnelli: oh, it also has blockhash
< jl2012>
sorry
< jonasschnelli>
Yeah. Just dbl checked: "" \"blockhash\" : \"hash\", (string) the block hash\n""
< jl2012>
thanks everyone......at least I find a way to do it without customized API....
< Chris_Stewart_5>
Is there a policy when RPC interfaces change for bitcoind? Does it change for minor version bumps?