< wumpus>
luke-jr: the only reason that the change to num files was acceptable is because leveldb, at least the version in our tree, doesn't actually keep more files open (on platforms with mmap) see also https://github.com/bitcoin/bitcoin/pull/12495#issuecomment-377228329
< Jmabsd>
> sipa, right and when getting a disassembly printout in Bitcoin Core and related tools, those 20B:s are printed in normal order
< Jmabsd>
What about pubkey hashes (20B), pubkeys (32B) and signatures (64B) - are those printed in normal or reverse byte order? so, I have a P2SH pubkey script, say. in there is a 20B hash of my redeemscript, right. when I use Bitcoin Core's script disassembly function, will it print that hash in byte or normal order? i mean there is an outer extent to what Core prints in reverse order - for instance, binary transaction dumps (in hex) are in
< Jmabsd>
wait, so Bitcoin has the tendency to print (256 & 160bit) hashes in *reverse* order, right - block hashes, transaction hashes and merkle root hashes.
< as1nc>
jonasschnelli, yes but i really want to incite people to txindex their chain so they can benefit the full spectrum of bitcoin capabilities. lightning for exemple require a txindexed chain right ?
< MaxHastings_>
jonasschnelli: Ah I see I did not know bloom filters were ineffective for keeping user privacy. Are there any other missing vulnerabilities on that page other than privacy concerns? I was told on Bitcoin slack that the SPV client could be tricked to change its consensus rules by malicious nodes.
< jonasschnelli>
But its OT in this channel... so use #bitcoin
< gmaxwell>
AFAIK, it just works. It's kinda limited though because none of them resist traffic analysis and it's SUPER easy to identify bitcoin traffic with traffic analysis. :)
< gmaxwell>
wumpus: I think jonasschnelli suggests that we have support for these obfscuated transports without putting tor in the middle. I think we already do (in fact, I used one of the ones made for tor to bridge bitcoin nodes last year when we were worried about china blocking bitcoin... before later changing to an icecast stream so that it wouldn't be identifyable by traffic analysis)
< jonasschnelli>
The idea is to make Bitcoin work in DPI env in case it would get blocked,.. like China or Iran
< wumpus>
integrating tor's code into bitcoin core is not a good idea
< sipa>
how is it bitcoin specific, or what do you specifically propose?
< wumpus>
looks like a layer violation to worry about this? or do you want to make tor tunnel over bitcoin instead of the other way around?
< _flow_>
What can I do to get https://github.com/bitcoin/bitcoin/pull/13621 merged? It improves the, currently terriable, configuration mechanics of bitcoind a bit (while still not perfect) and re-enables a recently disabled test.
< gnappuraz>
thx for the answer. But let's assume that I have to add a new file that if meant to be use by both bitcoind and bitcoin-qt, the rationale behind putting it in COMMON vs UTIL should be: if used by bitcoin-cli then UTIL, otherwise COMMON (since bitcoin-cli links UTIL but not COMMON)?
< jonasschnelli>
gnappuraz: these are semi-independent modules. Those modules get linked for the different binaries we have (bitcoin-tx, bitcoin-cli, bitcoind, bitcoin-qt).
< Jmabsd>
derp derp, ok Bitcoin Core gonna kick in more reverse order hex conventions in the ecosystem over time, he he. however users won't care really about those.
< achow101>
(in a modification of Bitcoin Core)
< Jmabsd>
gmaxwell: exactly, Bitcoin Core's HDwallet seeds are 160bit and presented in reverse order, yes
< Jmabsd>
some other hash, even if it relates to the bitcoin protocol, then normal order is fine
< Jmabsd>
achow101: exactly. so all over the whole Bitcoin ecosystem, those three values (block HASH, transaction HASH and merkle root HASH aka ID whatever), when you're dealing with hex representation, it must ALWAYS be in reverse order bc otherwise you'll screw up
< achow101>
Jmabsd: fun fact, bitcoin core does exactly that
< Jmabsd>
so we can only understand that transaction id:s and merkle root id:s (and in Bitcoin Core's case also hd wallet seeds) got this reverse byte order,
< achow101>
Jmabsd: Bitcoin Core checks that if the block hash is bigger than the target, it should fail, hence the return false that follows
< Jmabsd>
achow101: the "nBits" value, which is copied from the block header and is applied in PoW here https://github.com/bitcoin/bitcoin/blob/master/src/pow.cpp#L80 : "arith_uint256 bnTarget;bnTarget.SetCompact(nBits, &fNegative, &fOverflow);" , it's in the Bitcoin-internal 32bit floating point format right??
< Jmabsd>
wait, hashes are such are binary/byte concepts so the whole idea of relating to them as comparable numbers, is a concept that Bitcoin adds on top of them, right? the inventor of SHA256 never related to hashing as SHA256(byte string) => 256bit integer which may be little or big endian, right?
< Jmabsd>
dcousens: aha. Bitcoin Core would print the merkle root in reverse order. anyhow right.
< Jmabsd>
(which is some Bitcoin transaction for some purpose)
< Jmabsd>
for instance, for symmetry with Bitcoin, if you make a Dcousens Bitcoin Transaction format, then obviously its hash should be printed in reverse order.
< Jmabsd>
dcousens: yeap i see. i'm trying to make just a bit of sense here, so that, if you make a Dcousens Protocol to do some Bitcoin-related stuff, and you have some structure in your protocol that you hash and then give to people, in what order should it be.
< Jmabsd>
that makes sense yes. what code in Bitcoin Core compares two uint256:es, which would be used for sorting?
< gmaxwell>
21:30:21 < gmaxwell> block hash is interpreted as a 256-bit little endian number for target comparison as luke says, so then bitcoin needed to print it that way too, otherwise the 0s would have been on the right which would have been confusing... and the same code was then used to print other hashes.
< jonasschnelli>
Maybe they don't fall on bitcoin/bitcoin travis due to the exta CPU cycles we bought
< sipa>
airplanemod: bitcoin core is primarily developed on unix systems (and the windows binaries are produced through cross compiling on linux)
< sipa>
setting up a development environment isn't very bitcoin core specific, so i wouldn't say is on topic here
< airplanemod>
Hey guys, I would like to know if it is possible to set up a development environment for bitcoin-qt and/or bitcoind that will allow me to set breakpoints and step through live code as it is running? I cannot seem to find a tutorial or any good starting point for how to do this. I have been testing my own edits in a simple text editor and using gitian builder to compile and test using
< Jmabsd>
(yes I totally see that Bitcoin Core or the Bitcoin protocol never represent data in reverse order internally - it's only a human representation thing)
< gmaxwell>
block hash is interpreted as a 256-bit little endian number for target comparison as luke says, so then bitcoin needed to print it that way too, otherwise the 0s would have been on the right which would have been confusing... and the same code was then used to print other hashes.
< gmaxwell>
Jmabsd: the point is that bitcoin's UI uses the same stuff to print all these 256 bit hashes.
< Jmabsd>
gmaxwell, yes interesting, i see clearly in Bitcoin Core and other implementations that indeed there is no reversing whatsoever, no "endianness" or the like here. you are right that the code in the wild must have been mitigating some kind of user-facing hex representation quirk.
< gmaxwell>
Bitcoin "UI" hashes are backwards for path dependant but otherwise explicable reasons...
< gmaxwell>
Jmabsd: Sipa already answered this question previously. The bitcoin protocol uses sha256 exactly as it is.
< Jmabsd>
luke-jr: is any particular byte ordering or "endianness" to the SHA256 used here, i've seen some Bitcoin merkle node calculation use byte-reverse logics
< Jmabsd>
sipa, two quick off-topic but conceptually related questions, as general guidance for design of any new signing protocols, would you suggest using a single hash instead of double, or could it make sense to recycle Bitcoin's double-hash for symmetry with Bitcoin? also for guidance for other protocols, if you're going to do a salted hash of some message e.g. SHA256HMAC(protocolmessage,"SIPAPROTOCOL"), then for extension
< sipa>
bitcoin basically everywhere either uses ripemd160(sha256(x)) or sha256(sha256(x))
< Jmabsd>
and so Bitcoin is really only dealing with the "contract endorsement" structures, and hence in the real world those are 72B+1B sighhash=73B max.
< Jmabsd>
sipa: ah right, so, the zoomed-out situation in Bitcoin is that DER situations are in sigscript, redeem script (part of sigscript right) and P2WSH witness script (input's last witness element), only.
< sipa>
every signature in bitcoin script has a sighash byte
< Jmabsd>
Wait, in what situations are sighash bytes not added vs. added, to ecdsa signatures in Bitcoin? With a sighhash byte the DER signature max size is 73B (if low-s, 72B) whereas without sighash byte the max size is 72B (if low-s, 71B).
< Jmabsd>
wait, where are Bitcoin Core's logics to actually enforce little-endian serialization - say I run Bitcoin Core on an ARM.. is this the line that will cause the big endian (native to my computer) to little endian (serialization form) conversoin: https://github.com/bitcoin/bitcoin/blob/master/src/serialize.h#L89 ?
< Jmabsd>
sipa, the SHA256 specific about Bitcoin is why you need to byte-reverse most(all?) SHA256 libraries' results within Bitcoin merkle tree merkle node calculations, right?
< sipa>
(SHA256 internally uses big endian, but that's black box - it's a function that takes as input a byte array and produces another byte array from bitcoin's pov)
< sipa>
everything in bitcoin uses little endian
< Jmabsd>
Where is Bitcoin Core's serialization and deserialization code for values in various structures such as transactions, block headers, and protocol data? This code shows the endianness used
< wumpus>
Zenton: better ask in #bitcoin, this is a developer channel, you'd probably want to mention your perceived threat model for 'secure enough' as well
< sipa>
this discussion should probably move to #bitcoin
< unixb0y>
sipa: I run bitcoin-qt tbh :P
< gmaxwell>
Unfortunately, Bitcoin actually makes quite full use of the computer and so if its flaky at all it'll croak out. deleting blk and rev files is never going to help anything.
< unixb0y>
HI guys, I have a little issue with the initial Bitcoin Core setup.
< achow101>
oh, nvm, i see what you meant by bitcoin-qt being the non-deterministic one
< wumpus>
bitcoin-qt only suggests to me it's another non-determinism thing with the qt tooling, maybe file ordering or date/time in metadata in the qrc archives
< ken2812221>
The difference is on bitcoin-qt, I've compared it before but I deleted the docker one.
< promag>
in 13501, I was puzzled to know why bitcoin-cli worked differently than AuthProxy
< kevink>
I'm using Bitcoin Core's API and am wondering whats the difference between `duplicate-inconclusive` and `duplicate` when calling `submitblock`. I'm just trying to verify that a block exists in the blockchain and `submitblock` seems like the simplest way to do that.
< gmaxwell>
https://github.com/bitcoin/bitcoin/pull/13657 I don't understand this change. Coinbase transactions can be conflicted, by being orphaned block. I guess GetDepthInMainChain doesn't return negative in the case, but I think it probably should.
< Jmabsd>
sipa: exactly, i'm rading the relay code and can't find it too. so i was thinking maybe that bitcoin.org article is bss*ahem*incorrect*ahem*obsolete.
< Jmabsd>
"Bitcoin Core 0.12.0 defaults to relaying and mining null data outputs with up to 83 bytes with any number of data pushes, provided the total byte limit is not exceeded. There must still only be a single null data output and it must still pay exactly 0 satoshis."
< Gnappuraz>
Hi, I was going through the bitcoin documentation but I can't find the rationale behind the fact of using the prevTx.scriptPubKey in the place of curTx.scriptSig when creating the signature
< waiting2compile>
hi, could someone walk me a bit through running Bitcoin's tests? I tried following the instructions in the README.md, though I suspect there's some extra setup that needs to be done that's missing, since running the command just gives make errors and the like (pardon me, I'm new to this and am trying to fix one of the good first bugs :))
2018-08-13
< nmnkgl>
MarcoFalke: Could you point me to anything would prevent bitcoin core nodes from having crazy everlasting loops in case when there is a ring of whitelisted peers? In regular mode the loop will break at least because of using INVs. That may be very unlikely case though.
< gmaxwell>
I totally agree that wallets shoudl be batching and whatnot, but consider: we don't even have a friendly way to do that... There is no dohicky in bitcoin core where you can queue a payment, have it draft it, but not send it, waiting for either more payments it can be bached with, timeout, or shutdown trigger.
< itaseski>
i wasn't able to find anything bitcoin specific ...
< devmob>
hi, I'd really like to know how bitcoin does gossip, like how the gossip protocol is implemented
< reald0ff1>
well thanks for the answer. I think I will try to contact bitcoincore.org via email and bitcoin.org also via email to the website maintainer. Maybe some of them could provide me stats
< harding>
reald0ff1: I don't know if anyone has that information for BitcoinCore.org, sorry. In addition, the binaries can also be downloaded from Bitcoin.org (maintained by a different team) or via a torrent (with optional magnet URI) that contains the binaries for all platforms.
< reald0ff1>
Can someone please provide me download stats (or at least share in %) of bitcoin core, regarding the different platform versions (Win, Linux, OSX, etc) ?
2018-08-10
< gribble>
https://github.com/bitcoin/bitcoin/issues/13930 | Better explain GetAncestor check for m_failed_blocks in AcceptBlockHeader by Sjors · Pull Request #13930 · bitcoin/bitcoin · GitHub
< sipa>
this channel is mostly about developmemt of the bitcoin core software
< sipa>
Jmabsd: i think your questions are probably better suited for bitcoin.stackexchange.com
< Jmabsd>
are we in a place today where you can assume that every relevant participant in the Bitcoin ecosystem will be fine with Bitcoin transactions with version = 2?
2018-08-09
< sipa>
2018-08-09T18:55:00.301077Z receive version message: /bitcoinj:0.14.7/Bitcoin Wallet:6.28/: version 70001, blocks=535894, us=127.0.0.1:8333, peer=414
< sipa>
and your ISP's cost, and the bitcoin exchange rate?
< Jmabsd>
aj: ah, so Bitcoin Core nodes will accept a nonstandard tx when it's in a block, but it will not accept them in gossiping, right?
< Jmabsd>
Where in Bitcoin Core's code are the constraints on what scripts are allowed, implemented? E.g., gossiped transaction's lock scripts must be either of P2PK, P2PKH, multisig, nulldata or P2SH in order to be considered valid and be gossiped?
< MarcoFalke>
githubs send from:"notifications@github.com" to:"bitcoin@noreply.github.com" with cc:"comment@noreply.github.com" and cc you
< harding>
MarcoFalke: to ping privately, the bot could just pull the email address of the committer from the most recent commit message and email them. Alternatively, the bot could open an issue on the repository they're making their PR from, e.g. github.com/harding/bitcoin/issues/new
< gribble>
https://github.com/bitcoin/bitcoin/issues/13808 | wallet: shuffle coins before grouping, where warranted by kallewoof · Pull Request #13808 · bitcoin/bitcoin · GitHubAsset 1Asset 1
< gmaxwell>
https://github.com/bitcoin/bitcoin/pull/12495#issuecomment-367815005 < eklitzke suggests increasing leveldb's mmap limit here. ossifrage also apparently did that and reported it fixed his problems, though his specific problems are likely better fixed by switching to poll.
< gribble>
https://github.com/bitcoin/bitcoin/issues/13789 | crypto/sha256: Use pragmas to enforce necessary intrinsics for GCC and Clang by luke-jr · Pull Request #13789 · bitcoin/bitcoin · GitHub
< ossifrage>
mmap() is a great way to generate a very large amount of write pressure, but it seems like most of the leveldb use in bitcoin has a very low change rate
< gribble>
https://github.com/bitcoin/bitcoin/issues/13662 | Explain when reindex-chainstate can be used instead of reindex by Sjors · Pull Request #13662 · bitcoin/bitcoin · GitHubAsset 1Asset 1
< gribble>
https://github.com/bitcoin/bitcoin/issues/13676 | Explain that mempool memory is added to -dbcache during IBD by Sjors · Pull Request #13676 · bitcoin/bitcoin · GitHubAsset 1Asset 1
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #13247: Add tests to SingleThreadedSchedulerClient() and document the memory model (master...scheduler-tests) https://github.com/bitcoin/bitcoin/pull/13247
< bitcoin-git>
bitcoin/master cbeaa91 Jesse Cohen: Update ValidationInterface() documentation to explicitly specify threading and memory model
< bitcoin-git>
bitcoin/master b296b42 Jesse Cohen: Update documentation for SingleThreadedSchedulerClient() to specify the memory model
< bitcoin-git>
bitcoin/master 9994d01 Jesse Cohen: Add Unit Test for SingleThreadedSchedulerClient...