< GitHub136>
[bitcoin] dcousens opened pull request #8332: semi trivial: clarify witness branches in transaction.h serialization (master...patch-1) https://github.com/bitcoin/bitcoin/pull/8332
< gmaxwell>
dgenr8: uh. you realize that when you have .5 or more non-honest miners sustained, the existance of an honest blocks is purely their choice, right? So that graph doesn't make a lot of sense.
< gmaxwell>
yet again you have another negligently wrong security analysis, where you report 50% as absolutely safe, when in fact it's a guarenteed failure (assuming dedicated attackers)
< dgenr8>
indeed, a 51% attack is a far greater threat than a blocktime slowdown, you get it
< gmaxwell>
what? your analysis is just obviously broken on its face.
< gmaxwell>
it claims that 80% dishonest mining is unable to slew the time.
< gmaxwell>
This is obviously untrue, so you've make some obvious mistake-- if you have any result that isn't 100% at 50% hashpower.
< gmaxwell>
And, what the heck is "a 51% attack" -- Bitcoin exists as a confluence of incentives, one part of it is that even a user who buys up large amounts of hashpower for a brief period only gains the ability to reorganize recent transactions, which is difficult to profitably exploit at any scale-- which users can protect themselves from by waiting for additional confirmations. Under the security mod
< gmaxwell>
el change that you propose (and incorportated into your software without an adequate disclosure because you (based on the prior conversation) weren't actually aware of the implications, miners that achieve a large amount (unclear how much but a majority is clearly sufficient) can add huge amounts of coins to themselves without any chain reorg and without even making a purchase.
< gmaxwell>
(re obviously didn't understand, I'm referring to your prior claim that 100% hashpower and control over the user's clock was needed.)
< dgenr8>
you read that statement completely wrong
< gmaxwell>
K. Did I misunderstand that this was shipped out in XT without a release note pointing out the security model change; to a user community that obsesses over even minor softforks as taking away validation?
< gmaxwell>
am I misreading your graph showing that a miner with 79% hashpower has a non-unity probablity of success at moving the timestamp back?
< dgenr8>
XT, unlike core, would warn loudly as that 51% attack slowed the generation rate
< gmaxwell>
Did I miss the fact that this whole design is just an insecure rip off of the proposal from this channel to use 30 days of _work_ at the current tip, as a gate on signature checks?
< gmaxwell>
dgenr8: there is no need for the generation rate to slow significantly enough to trigger any warning that doesn't false positive constantly.
< dgenr8>
the reason XT exists is that a few people cannot abide the choices you are making. it's too bad. for a while, folks worked together pretty well.
< dgenr8>
you view the loss of talented people like gavin, jgarzik, and hearn as positve, and seek to utterly discredit others. but i'm OT
< gmaxwell>
dgenr8: You are going to get some people robbed blind with this kind of sloppyness; it's a risk to our entire industry.
< gmaxwell>
I don't have to do anything to discredit you (not to mention other people)-- you do so _yourself_; by shipping software to users that you don't even understand and being continually antagonistic to people who would otherwise help you avoid danger.
< dgenr8>
Perhaps try again to manipulate the timestamp on testnet. Any hashing that sets timestamps correctly has a huge advantage though.
< gmaxwell>
dgenr8: you didn't even notice the successful attack on testnet against classic then, I guess.
< dgenr8>
i did see it was pushed it back a few days
< gmaxwell>
A few days ago you seemed to be arguing that using the _miner provided_ timestamps on blocks to decide to bother validating the block or not would require 100% hashpower and control of the users local clock to exploit-- functionality you added without even adding a _release note_ to Bitcoin XT. Your response to this being pointed out was to again repeat accusations that I backdoored your softwa
< gmaxwell>
re. After having your mistaken understanding thrust on you; you've gone away and come back with another obviously broken misunderstanding; again radically overstating the security of this 'optimization'.
< gmaxwell>
The only reason I even knew that XT and classic had incorporated this ill advised security change is because of people bragging about how much faster it synced. Yea, it's faster when you don't check 95% of the signautures, blindly...
< gmaxwell>
So if you think you're going to make a war for node operation a war about who can make the most insecure software-- the stuff with the most tail risk of total failure, then hell yes I'm going to call that out. And you should be thankful that I've taken the concerns directly to you, and not-- like other people in your community-- gone straight to the press with them.
< gmaxwell>
(Especially when it's so easy to demonstrate the exploit on a live network)
< dgenr8>
hmm. i think this whole display is for the benefit of third parties. anyhow i disagree on the degree of risk, but that's not to say the regular checkpoints couldn't come back or that the default selection could even change. but that belongs to a rational discussion we're not having
< gmaxwell>
Fair enough.
< dgenr8>
my concern with your thin blocks change was not that it made it to XT briefly. it was that you removed the thin blocks functionality from core users.
< gmaxwell>
I'm sorry for being such a dick on this. I do really think that the blockheader based skipping validation is a race to the bottom-- who can provide the worst security (for the fastest performance), to hell with the consequences (and no one adequately disclosing it). It doesn't help that a much better way of doing the same thing was already proposed in Core; and to me it seems like pure NIH to i
< gmaxwell>
mplement a much riskier version.
< gmaxwell>
dgenr8: I had no idea that it had any impact there, hell, the change was actually motivated by a complaint you made about the behavior (which, since you didn't disclose, I didn't know it was due to the prior limit of 1000 breaking mike's thinblocks). I didn't even initially use the bloom filter until wumpus and pieter beat me down on the memory usage. And ultimately since there was _no_ reliabl
< gmaxwell>
e mechenism to request transactions that were already in a block (an observation we made on that approach in 2013), mike's way of doing it couldn't have been reliable.
< gmaxwell>
I also explained this previously, but you seem even more insistant on assuming bad faith on my part than I am on seeing you as a fool. :)
< gmaxwell>
Seriously, if I wanted to 'backdoor' your software, the avenues available are far more profound-- your response has made me profoundly uncomfortable with the continued existance of Bitcoin XT-- you'll continue to copy code from Core, and when the inevitable errors that happen from interactions which I could never have anticiapted; you've shown that you'll accuse me of unethical or even criminal
< gmaxwell>
conduct.
< gmaxwell>
s/blockheader based/blockheader timestamp based/ to be more clear. the earlier proposal of using total work instead of timestamps, is much more appealing.
< dgenr8>
whoa slow down ... the map size problem I found had nothing to do with thin blocks, which is why i was eager to copy your fix for it
< gmaxwell>
dgenr8: I thought (after trying to figure out why you even copied the change) the reason you commented on the map size is because with thinblocks the small map meant the far end had no idea that you had most of the transactions you already had, then sent redundant stuff.
< dgenr8>
no, i found it while looking at some other core PR
< gmaxwell>
then why did you even assume I was trying to screw up your thinblocks stuff? I don't believe I had _any_ idea XT was even screwing with that at that time.
< gmaxwell>
(I wouldn't have assumed it because the patch mike had was so obviously junk-- handling it in the ping handler, no way to cope with missing transactions except praying that they were still in the relay cache)
< dgenr8>
if you recall that effect was found independently weeks later by Peter Tschipper
< dgenr8>
its all irrelevant now anyway
< gmaxwell>
in any case, indeed it broke the thin block stuff completely; then you merged it, and released a new version with it and the initial release of that thinblock stuff--- meaning you hadn't even tested it for basic functionality; which is basically terrifying from my perspective.
< gmaxwell>
You can't deny that the commit message was scream loud totally clear about _exactly_ what the change did.
< gmaxwell>
and so how the hell can I defend myself against bad testing on your part? I made a very very clearly described change, which you didn't comment on, integrated so it broke your stuff totally, then claimed misconduct on my part. Thats seriously bad mojo.
< dgenr8>
it went to master, not a release. please don't put 'backdoor' in quotes as I never said that word and as I said, that was not my concern
< gmaxwell>
Though indeed, I guess you only use the words "your sneak attack" and not "backdoor" yourself.
< gmaxwell>
and "Here we have Blockstream actively fighting scaling improvements" (and not to mention misrepresentation the that change was to avoid a one in a million false positive, when the chance happens for every txn indpendantly, so after seeing just 10k txn the chance is 1%.)
< gmaxwell>
XT v0.11E appears to contain the setinventory known and 'thinblock' change; but now looking at it I see you left it probablistically screwing over spv clients.
< eragmus>
gmaxwell: Why are you wasting your precious time with a dgenr8 like him? :) Or, at least, maybe charge him and the others by the minute for this consulting.
< gmaxwell>
I wonder how many bitcoins will be lost by people throwing out wallets that look empty but arn't. :(
< eragmus>
dgenr8: Btw, you should be careful before making everyone laugh so much -- "the loss of talented people like gavin, jgarzik, and hearn" -- I nearly choked. I might have to bill you for any potential hospital expenses.
< gmaxwell>
eragmus: (1) because software that silently downgrades security then promotes itself as faster is a risk to the continued viability and value of Bitcoin, which I care about. .. and because he attacks me by claiming that I've acted unethically, which if left unchallenged ends up recycled as accepted truth in places like the NYT where it can cause me lifelong harm.
<@wumpus>
PSA: this channel is about bitcoin core development, XT and classic and certainly non-technical drama around them are very off-topic
< phantomcircuit>
indeed wumpus is right, possibly this could move to #bitcoin
< dgenr8>
gmaxwell: breadwallet already detects the condition and corrects it by reconnecting
<@wumpus>
phantomcircuit: in what world is 31.87ms slow, I mean, I've seen logs in where validation took that *per txin* :)
< phantomcircuit>
wumpus, in a world where i can mess with settings and have an average block validate in 30ms
< phantomcircuit>
thanks to my 8GiB sigcache and 16GiB dbcache
< phantomcircuit>
(and a hack that loads the entire utxo into cache on start)
< gmaxwell>
wumpus: he's optimizing mining overheads, some of them are harder to address than bitcoind behavior. :) (e.g. cannot decrease the radius of the earth :P )
<@wumpus>
phantomcircuit: anyhow to say more about it we
<@wumpus>
'd need detailed profiling of that step
< phantomcircuit>
gmaxwell, im working on that, i have a plan to drill down and around the mantle
< phantomcircuit>
i only need a few trillion dollars
< gmaxwell>
/msg phantomcircuit has the patent issued yet for the neutrion based transciever yet?
<@wumpus>
gmaxwell: yes can only get as close as possible to the physical limit
< gmaxwell>
wumpus: there are a number of things that aren't the physical limits but are still harder to speedup than core, also at least as long as validation is in the relay critical path, any part of validation adds up as the node count increases. Though thats probably better fixed by getting validation out of the critcial path for relay to other full nodes.
<@wumpus>
gmaxwell: what kind of things, optimizing the P2P protocol? network infrastructure?
<@wumpus>
as long as any part of validation is the bottleneck, optimizing that is still part of 'speeding up core'
<@wumpus>
if validation is out of the critical part all that is left is the network, where the random P2P network isn't the most efficient structure for propagating something around the world quickly
<@wumpus>
(but that's what the relay network is for)
< sipa>
phantomcircuit: it's good to see that part of the logic becoming measurable :)
< sipa>
phantomcircuit: it would be nice if you could see if the two PRs i referenced on your issue affect that number
< michagogo>
Has anyone looked into packaging Bitcoin Core into a snap?
< michagogo>
(Ubuntu's new packaging format)
< michagogo>
I haven't gotten a chance to look at it closely (and I don't know if I'd really understand it), but it sounds like it's an Ubuntu-supported way for devs to package apps in self-contained packages that can update independently of the distro
< michagogo>
Which sounds like it would solve a lot of the problems we've had with Bitcoin being packaged in the Ubuntu repos
< michagogo>
(i.e. it _might_ let us sanely get Bitcoin Core into Ubuntu)
< sipa>
sounds interesting
< jonasschnelli>
sounds good... the current PPA is maintained by thebluematt, maybe he's interested
< xinxi>
I've herd there is slack group for Bitcoin core dev. Who can tell me the URL?
< btcdrak>
xinxi: slack.bitcoincore.org
< eragmus>
xinxi: https://slack.bitcoincore.org -- However, it is not really focused on Core development. It's a general purpose community with nearly 2,000 members who discuss all manner of topics.
< btcdrak>
but it's more a community slack.
< xinxi>
thank you.
< xinxi>
Oh, messages sent on slack cannot be seen here.
< Chris_Stewart_5>
Is there some weird semantics with TransactionMessage on the p2p network that are different than BlockMessage? Besides switching MsgBlock to MsgTx
< Chris_Stewart_5>
I keep getting a NotFound message response, when I know the tx exists
< sipa>
Chris_Stewart_5: you can't just request any transaction
< sipa>
only things that have been advertized to you
< sipa>
(reasons for this are 1) there is no full index for all transactions in the chain 2) there is a privacy leak if you allow peers to check whether you have seen a particular transaction already)
< Chris_Stewart_5>
so what does -reindex allows you to easily search by header id then? And can you elaborate more on 2)? Does that allow them withold honest txs from you if you are being sybil attacked?
< sipa>
Chris_Stewart_5: the network tries to hide where a transaction originates
< sipa>
for that, transactions are relayed with random delays between them, and in different order for different peers
< sipa>
if i could query whether you have a transaction that i know about, but you haven't told me about, i can bypass that mechanism
< sipa>
Chris_Stewart_5: -reindex just throws away the database and builds a new one
< Chris_Stewart_5>
Sorry i meant -txindex
< sipa>
yes, that's for local consumption
< sipa>
not exposed to the network
< Chris_Stewart_5>
Ok. So if I am understanding you correctly we allow arbitrary block querying because 1.) we need it to sync nodes on the network, 2.) You can't tell where a tx originates from once it is included in the block
< Chris_Stewart_5>
What file is the code in wrt to transaction propogation? main.cpp? net.cpp?
< Chris_Stewart_5>
I interpreted it as you build a getdata message, send it and then get a tx message response
< sipa>
it could be clarified
< sipa>
but it's not wrong that a tx is sent as response to getdata TX
< sipa>
but i agree it's a bit misleading
< Chris_Stewart_5>
sipa: With the caveat being that the node has already broadcasted the txid to that node requesting the tx with a getdata right?
< sipa>
or through the bip35 mempool command
< Chris_Stewart_5>
gotcha, thanks.
< sipa>
also, after relay, you only have a finite amount of time to getdata
< sipa>
i think 15 minutes by default
< Chris_Stewart_5>
Seems like that could have interesting consequences if the hash rate where to suddenly drop significantly. Tx propogation could stagnate couldn't it?
< sipa>
tx propagation has nothing to do with blocks
< Chris_Stewart_5>
True, I guess the scenario I was envisioning in my head would only happen if nodes were constantly being shut off and turned on at an alarming rate
< bsm117532>
My transactions spending segwit outputs are being rejected by sendrawtransaction...
< sipa>
what code?
< bsm117532>
Do I understand correctly that it's normal behavior for the segwit input scripts have a leftover element on the stack?
< sipa>
indeed
< bsm117532>
64: non-mandatory-script-verify-flag (Script evaluated without error but finished with a false/empty top stack element)
< sipa>
that likely means your signature is incorrect
< bsm117532>
Hmmm ok
< bsm117532>
I tried to create a spend of a P2WPKH transaction, with the signature and pubkey as the two elements in the witness data. But signrawtransaction is giving me a txinwitness that contains a signature and a 65 byte-blob. Is this second value an uncompressed pubkey or something?
< bsm117532>
Can someone explain the difference in what sendrawtransaction is putting as the second element of the witness data?
< sipa>
yup, that looks like an uncompressed pubkey
< bsm117532>
Are uncompressed pubkeys required? AFAICT that's the only difference between the two.
< sipa>
no
< sipa>
they're even discouraged and no modern bitcoin core wallet will use them by default
< sipa>
but if you have a very old wallet file or have imported uncompressed keys, there is no choice
< sipa>
*won't use them by default
< bsm117532>
Ok so if sendrawtransaction is using uncompressed pubkeys, it could be a bug, and if segwit script validation is rejecting compressed pubkeys, it could be a bug...I'll dig.
< sipa>
i can guarantee you that compressed pubkeys work
< sipa>
i suspect that you're just signing the incorrect signature hash
< bsm117532>
That's entirely possible.
< sipa>
(that's just from experience of other people seeing fail there)
< bsm117532>
Ok I'll dig there first. But I'm also working on a wallet-functionality patch for core. My node shouldn't be generating uncompressed pubkeys, though I do run --with-incompatible-bdb, other than that all keys should be new.
< sipa>
did you import keys, ever?
< bsm117532>
Hmmm...possibly.
< bsm117532>
Should I recreate my wallet then?
< sipa>
imported keys have a flag that says whether the corresponding pubkey is to compressed or not (otherwise the address is not well defined)
< sipa>
i wouldn't bother
< bsm117532>
Hmm maybe I just got unlucky with the chosen input.
< * bsm117532>
beats on BIP 143. I knew it was unlikely that my sighash was correct...
< * bsm117532>
sees now...BIP 143 isn't even remotely close to the pre-segwit algorithm.
< sipa>
nope
< sipa>
it's still more or less the same order of data, but all 'over all inputs' or 'over all outputs' things are first aggregated into an intermediary hash
< bsm117532>
In my defense, I wrote in the python-bitcoinlib PR that I didn't implement this yet! ;-)
< bsm117532>
In BIP 143, this line: if (!(nHashType & SIGHASH_ANYONECANPAY) && (nHashType & 0x1f) != SIGHASH_SINGLE && (nHashType & 0x1f) != SIGHASH_NONE) {
< bsm117532>
Isn't it simpler to just say: if(nHashType | SIGHASH_ALL) ?
< bsm117532>
errr.... if(nHashType & SIGHASH_ALL)
< sipa>
that would not be the same, i think?
< bsm117532>
I think it's the same unless there are more than 4 SIGHASH_* types
< sipa>
there are 6 combinations
< sipa>
but the sighash byte can have 256 values
< sipa>
all of which map to one of those 6 formulas
< bsm117532>
I see. I thought it was just a bitmap.
< bsm117532>
Reference implementation is easy to translate to python, I'll stop trying to "improve" it. ;-)
< bsm117532>
Sorry I hadn't realized this was so close to petertodd's repo.
< * bsm117532>
diffs.
< sipa>
i think it forked off a long time ago
< sipa>
it's a forked, stripped, and then extended version just for tests
< bsm117532>
makes sense.
< sipa>
but feel free to steal/donate code of course
< bsm117532>
Thanks, I'll attribute what I steal.
< midnightmagic>
sipa (or anyone really): When a node hears about a sibling block to its current valid head, it ignores it, correct? Or does it hold it until it discovers which fork the miners settle on?
< midnightmagic>
I thought it sits on it so long as it is also valid and does not represent less work than the current head?
< bsm117532>
midnightmagic: It's my understanding that it doesn't *relay* it, but does track its chain tip, in case its PoW becomes larger.
< sipa>
midnightmagic: we keep track of the headers
< sipa>
midnightmagic: but only actively try to fetch the blocks if they're overtaking our best chain
< sipa>
under certain circumstances we do accept unsolicited blocks
< midnightmagic>
sipa: can you give me an example of one of those edge cases? I don't need gritty details..
< sipa>
midnightmagic: not edge case; it's very intentional... for example, nodes connected to the (old) relay network would get blocks pushed directly to them
< sipa>
it would be silly to not accept those
< midnightmagic>
sipa: So an old-style block chatter is still accepted wholesale and validation proceeds (and then stored but still not considered canonical head until Y+1 indicates the fork has overtaken X.)
< sipa>
midnightmagic: well in almost all cases the new block is not a reorg
< sipa>
but we don't know that in advance
< bsm117532>
petertodd: I notice in python-bitcoinlib that CScript treats OP_0 as an empty data push b''. It's used as the segwit version number. Is there good reason to treat OP_0 as an empty data push over a small integer with value 0?
< sipa>
bsm117532: you can't change the semantics of the scripting language...
< sipa>
OP_0 by definition pushes the byte sequence ''
< bsm117532>
Is that changing the semantics?
< sipa>
yes
< sipa>
if you would compare it to the empty string later, it has to succeed
< midnightmagic>
sipa: Is the strict difficulty of the block hash now considered in terms of deciding validity of head? I know you guys were thinking of doing that instead of just the difficulty target of the block itself..
< sipa>
bsm117532: the script language stack only has one data type, byte array
< sipa>
some operators treat the elements as numbers, but that's not relevant
< bsm117532>
sipa: okay, I'm comparing to your bip141 which writes scriptPubKey as: 0 <20-byte-key-hash>, but obviously b'' <20-byte-key-hash> is equivalent.
< sipa>
no
< sipa>
it's OP_0 and then a 20-byte push
< sipa>
oh, that's what you mean
< sipa>
why can't you use OP_0 ?
< bsm117532>
Yes. python-bitcoinlib treats it as an empty byte array push, followed by a 20-byte push.
< sipa>
that's correct
< bsm117532>
It doesn't give me OP_0. It decodes OP_1...OP_16 but does not decode OP_0.
< bsm117532>
This is to do with the __iter__ method of CScript, which parses it.
< sipa>
that sounds like a bug
< bsm117532>
That's why I asked. ;-)
< sipa>
but i don't know python-bitcoinlib
< bsm117532>
Maybe petertodd will happen by ;-)
< bsm117532>
There are two iterators, a "raw" iterator, for people who want to tell the difference between PUSHDATA* and a "cooked" iterator which is supposed to be "intuitive" or something. I'll put the difference into the "cooked" iterator.
< bsm117532>
so now I get: CScript([OP_0, x('a87e6d173860ff1dfc8849f2638182aa36058389')])