< gmaxwell>
phantomcircuit: does that actually work?!
< gmaxwell>
add docs.
< phantomcircuit>
gmaxwell, yeah it does
< phantomcircuit>
i had completely forgotten about the fRelay flag
< dcousens>
phantomcircuit: its cool tbh haha
< dcousens>
conceptually, phantomcircuit I feel like you could separate mempool from bitcoind with -blocksonly :P
< dcousens>
phantomcircuit: also, would it be worth just dropping the transactions before they even get to AcceptToMempool?
< dcousens>
or would advertising you don't care about txs be a BIP requiring protocol change?
< dcousens>
gmaxwell: getdata being dropping the transactions before ATM? :P
< gmaxwell>
jinx I guess. :)
< dcousens>
haha
< gmaxwell>
but yea, I think he should change it to not getdata any announcements instead from non-whitebound peers, and leave the mempool alone.
< dcousens>
gmaxwell: agreed, most likely the mempool would always be empty
< dcousens>
but that is intended
< dcousens>
(obviously)
< gmaxwell>
Right but then you could still use this node to e.g. relay transactions created by an offline wallet behind it and such.
< dcousens>
gmaxwell: or as I was thinking, have multiple relay nodes with decent bandwidth, but low memory, backed by a [lower] bandwidth blockchain node
< dcousens>
assuming you had a service big enough where that is warranted anyway
< dcousens>
hmmm, though, its the mempool that takes all the memory anyway haha
< dcousens>
well, it'll be interesting to play with for sure
< phantomcircuit>
dcousens, im dropping them in AcceptToMempool to prevent leaking which transactions are yours
< phantomcircuit>
im not sure i did a thorough job of that though
< gmaxwell>
phantomcircuit: nah, don't do that. Turning off wallet broadcast (already an option) is how thats achieved.
< dcousens>
gmaxwell: what about if it was submitted via RPC?
< gmaxwell>
Then we want to relay it.
< dcousens>
gmaxwell: sure, but, that seems counter to the point of preventing leakage
< gmaxwell>
blocksonly shouldn't refuse to relay transactions when you've specifically asked it to so. :)
< gmaxwell>
dcousens: if you don't want it to broadcast a transaction then don't ask it to!
< dcousens>
gmaxwell: sure, but, you could say the same about using it as a wallet
< phantomcircuit>
gmaxwell, well then it's a one line patch
< gmaxwell>
phantomcircuit: I think not, you need to avoid getdataing if they ignore your flags.
< dcousens>
gmaxwell: I agree with you, just pointing out the logic didn't work with what phantomcircuit logic said "preventing leaking [of] transactions which are yours"
< gmaxwell>
dcousens: his current behavior will not relay a sendrawtransaction.
< gmaxwell>
I thought you were asking what would happen in my proposed change.
< dcousens>
dw, I was
< dcousens>
unimportant, semantics are clear either way ;)
< gmaxwell>
phantomcircuit: so I think behavior should be when set, you will set the flag (to suppress inv announcements), and ignore getdatas. But only from non-white peers. And then anything sendraw/wallet will still work like normal, and if you don't want that for privacy reasons..then disable wallet broadcast and don't use sendraw. :)
< gmaxwell>
I imagine post 0.12 we'll also make this behavior one of the things we do in response to running into bandwidth limits.
< gmaxwell>
(e.g. if you get too close to your limit, go into blocks only mode.)
< phantomcircuit>
gmaxwell, that reminds me, thoughts on having a global settings object? (something that ya know.. actually acquired a lock)
< gmaxwell>
Big lock protecting bag-of-assorted things isn't generally great.
< gmaxwell>
in any case, not needed here, auto-blocks only I think is out of scope for 0.12. It's a pretty big behavior change, and we'd have to do something about the transaction privacy loss. (hopefully fix that via a seperate private announcement mechenism)
< dcousens>
gmaxwell: by auto, do you mean default?
< dcousens>
oh, you mean auto bandwidth response
< dcousens>
nvm
< phantomcircuit>
gmaxwell, could equally be a separate lock on each of the settings
< gmaxwell>
phantomcircuit: right for settings that should chang at runtime the setting should be read in init and set in a datastructure with its own lock for further changes.
< phantomcircuit>
gmaxwell, well and most of them can just be atomic ints
< gmaxwell>
We are not yet using C++ new enough to have standard atomic types.
< gmaxwell>
And plain integers ARE NOT ATOMIC.
< jgarzik>
without help
< gmaxwell>
sure, you can use locks. Or custom asm or what not.
< jgarzik>
gcc & clang provide atomic intrinsics to give C/C++ level access to cmpxchg, ints with lock prefixes and more.
< jgarzik>
not C++ standard but it is available on all our platforms
< gmaxwell>
/usually/ one shouldn't be directly using atomics in any case except in performance crticial algorithims or when building lower level constructs to be used from other things.
< gmaxwell>
If the boost stuff is drop in for c++11 too than thats not so bad.
< jgarzik>
that's the idea
< gmaxwell>
I think we're already in a mode of only using boost features that aren't in C++11 if we have reason good enough to justify reimplementing them internally in any case. (e.g. mempool indexing)
< phantomcircuit>
gmaxwell, hmm so im thinking blocksonly should be undocumented as there are actually very few people who should be running such a thing
< phantomcircuit>
the privacy loss form running with blocksonly and walletbroadcast=1 is potentially huge
< dcousens>
phantomcircuit: that seems wrong
< dcousens>
maybe a compile flag to allow it
< dcousens>
or something
< wumpus>
just disable walletbroadcast by default if it's enabled?
< dcousens>
but not undocumented
< dcousens>
or that ^
< wumpus>
and emit a warning to the log
< wumpus>
PRIVACYWARNING: wallet broadcasting has been disabled, because any transaction you send while not relaying transactions will stand out like a sore thumb
< phantomcircuit>
wumpus, maybe... that's just what we need a more complicated tree of which settings are compatible :P
< wumpus>
if you are sure you want to enable this, use -walletbroadcast
< gmaxwell>
phantomcircuit: yea, you can imply walletbroadcast off for that case. We have handling like this for many options related to privacy.
< wumpus>
you call that complicated?
< sipa>
phantomcircuit: wallet broadcasting and whitelisting are already inherently a privacy leak
< gmaxwell>
though I think you are GREATLY overestimating the improvement relay has on privacy.
< dcousens>
wumpus: he just doesn't want to make it more than 1 or 2 lines
< gmaxwell>
I think either way (defaulting walletbroadcast off or on) is okay with me, though I would leave it defaulting on.
< phantomcircuit>
i uh dcousens *shhhh*
< wumpus>
'it is a already bad' is no excuse for allowing things to get silently even worse, hence the warning.
< wumpus>
that's the problem with privacy everywhere, people think well we have no privacy already anyway, so why not do this and this also, it's not like it could get any worse. Yes, it can get worse.
< sipa>
wumpus: agree, but to someone already listening on many nodes in the network, if they're connected to the original broadcaster too, they have a fairly good chance of finding you
< gmaxwell>
wumpus: I don't know if it makes it worse or not, it might-- so fair enough.
< sipa>
not relaying transactions otherwise just reduces the noticed; it doesn't fix the fact that information is being leaked
< wumpus>
gmaxwell: well if you never send transactions then suddenlly send one, that makes it extremely clear it's yours, doesn't it?
< gmaxwell>
sipa: research so far suggests that its currently about 100% effective if you transact more than once.
< gmaxwell>
wumpus: well with whitebind, perhaps not! (also if we start turning this on dynamically in the future e.g. when low on bandwidth) perhaps not. But fair enough.
< sipa>
s/noticed/noise/ i do not comprehend how i made that typo
< gmaxwell>
phantomcircuit: so go softset off wallet broadcast and log, see the other examples in init.cpp.
< wumpus>
I just don't like a setting that can silently make your privacy worse
< gmaxwell>
wumpus: yup easily avoided, I agree.
< sipa>
i'm fine with default off
< wumpus>
the other option would be to indeed not document the option
< wumpus>
but I think a warning + disable wallet broadcast by default is good enough
< sipa>
what about whitebind?
< wumpus>
what's with whitebind?
< wumpus>
you mean - do relay transactions for whitelisters?
< sipa>
whitelisted peers that relay transactions
< gmaxwell>
I think it'll be a nice bandwidth reduction feature for watching wallets. It's a >2x reduction in bandwidth (probably more like 3x typically) and also saves you from a number of dos attacks. :)
< gmaxwell>
I think it should relay for whitelisted peers. So you can use a node configured like this as a gateway to get your transactions to the outside world.
< wumpus>
gmaxwell: it is a great feature, but the catch is that you have to make sure you're silent yourself, as there is no noise to hide in. That's acceptable, but needs to be transparent.
< sipa>
how is whitelisting different from wallet broadcasting yourself?
< gmaxwell>
because you choose and configure whitelists, it's like setting broadcast=1
< sipa>
even if it doesn't disclose the ip of the actual sender anymore, they do know an ip address that the sender trusts
< wumpus>
right it's something you have to explicitly enable
< gmaxwell>
That could be a seperate setting. I've wanted that before, to control the forced relay for whitehosts.
< wumpus>
... which could come with a warning, of course
< sipa>
wumpus: then how is it different from walletbroadcast?
< wumpus>
sipa: my proposal is that walletbroadcast would be disabled by default if block-only
< wumpus>
whitelist already *is* disabled by default
< sipa>
hmm, ok
< wumpus>
walletbroadcast is enabled by default
< sipa>
ok, that is a fair point
< gmaxwell>
Whitelist is a bot too overloaded. We might want to add a whiterelay that controls the forced relay for whitelisted hosts, which defaults on, but is soft-off with blocksonly
< wumpus>
there are probably more ways that whitelisted peers can screw up your privacy :D
< gmaxwell>
s/bot/bit/
< wumpus>
they're sort of trusted
< gmaxwell>
yea, we already bypass mempool policy
< sipa>
wumpus: they are very trusted... but you yourself are also trusted
< sipa>
if you do a wallet transaction, you expect it to be relayed; if a whitelisted peer relays a transaction, you also expect it to be relayed
< wumpus>
the wallet may also rebroadcast old transactions
< wumpus>
that's the scary part
< sipa>
whitelisted peers will also rebroadcast old transactions
< wumpus>
even transactions sent to you
< wumpus>
ok -I feel this is not constructive anymore
< wumpus>
don't introduce features that may silently reduce user's privacy even more, that's all I ask
< sipa>
ok
< wumpus>
if there is warning, or a knob that has to be explicitly enabled, I'm ok with it
< gmaxwell>
yea, seems simple enough, softset off walletbroadcast when this is set.
< wumpus>
right
< wumpus>
and yes, whitelisted peers have to be careful, we should document that
< gmaxwell>
yea, we should document that already because they are already deanonymizing.
< gmaxwell>
(not even the primary reason I really don't like the whitepeer relay rules--)
< wumpus>
and yeah we could have an option to not relay transactions from whitelisted peers...
< gmaxwell>
(I mostly don't like them because I whitebinded the relaynetwork client and p2pool for obvious reasons, and then found they were blowing past my relay policy :-/ )
< sipa>
gmaxwell: i think the rebroadcasting is fundamentally the problem, because nobody but the owner of the transaction will rebroadcast
< sipa>
an alternative is peers syncing the mempool with each other...
< gmaxwell>
sipa: It's fairly busted without that, in fact.
< sipa>
so transaction owners don't need to rebroadcast randomly
< sipa>
it is... in theory you can end up with a cordon of nodes that already know your transaction between you and miners
< gmaxwell>
Right now on any given publically accessible node >=15 peers are spynodes that connect to everyone. They also use addr behavior to infer things about the non-listening peers behind them; and research shows high success rates for even single transaction origins. If you make a couple transactions in the current network you should assume your origin WILL be idenfied.
< gmaxwell>
To avoid this we need an actually private broadcast mechenism.
< wumpus>
yes
< gmaxwell>
and once we have that, it would work just as well for rebroadcast. :)
< wumpus>
we need a private transaction mechanism badly, but we already knew that I think :)
< gmaxwell>
and also would remove some of these discussions since then the p2p protocol would not have any privacy implications for full nodes (beyond running a node at all. :) )
< wumpus>
right, it would, but it doesn't affect our decisions right now because it's entirely hypothetical :)
< gmaxwell>
Just another benefit. It's really nice to not have to consider privacy implications for things; especially when the current state is bad enough that it can be hard to tell if you're making it worse. :)
< gmaxwell>
E.g. one of the problems with peer rotation, ... in privacy.
< gmaxwell>
er is privacy.
< sipa>
we just need separate networks for block relay and tx relay :)
< gmaxwell>
One of the problems with pruning ... is privacy (makes nodes more fingerprintable; so you can track nodes across tor and then correlate transactions they originate)
< sipa>
well... i guess the same arguments apply to mining actually, it's just much less data to analyse, but ideally you also don't want to be able to pinpoint block creators
< gmaxwell>
yadda yadda. All becomes more tidy if these things are seperated. In one sense you can see that privacy and partition resistance are actually in conflict.
< gmaxwell>
For privacy you want to tell your transactions to very few parties, hopefully not attackers. For partition resistance you want to talk to everyone. :)
< gmaxwell>
sipa: yea, though different problems; mining is very latency sensntive; tx broadcast is not.
< wumpus>
even dropping off your transactions to a random node over Tor is better than publicly broadcasting your transaction from your own node
< gmaxwell>
yea, but requires having a tor client. :(
< wumpus>
it does, having a tor client seems more and more a requirement for any kind of privacy unfortunately
< gmaxwell>
it's just a lot of code and exposure, hate having reason to mix things like that with other things that have critical security requirements.
< wumpus>
a tor bitcoin bundle would be pretty nice
< wumpus>
well at least it gives plenty of noise to hide in
< wumpus>
if you were to say, set up an onion routing network just for transactions, even partaking in it would still leak a lot information about you. Also, if you write it yourself doesn't make it any less 'code and exposure'
< gmaxwell>
It's true, for tx broadcast though, one could send a transaction through a high latency mixmaster like system entered via normal internet. And the daemon could just send an empty message an interior hop one ever couple minutes to hide the traffic. Doing the first hop over tor is superior, but I think not strictly needed.
< wumpus>
I agree that such a solution could theoretically be better, the problem as we've seen for years is that no one is implementing such a thing, and tor already exists
< gmaxwell>
wumpus: the client could be quite simple relative to tor (which at the very begining immediately needs a full SSL stack)
< gmaxwell>
Indeed.
< gmaxwell>
Actually someone contacted me recently about implementing such a thing, but I don't know if they will.
< gmaxwell>
(well they said they will, and asked if I'd be willing/able to review; but you know most things don't actually happen :) )
< gmaxwell>
wumpus: e.g. sending client for such a thing could encrypt a message, connect to a TCP socket, and send it over. Never reading a byte off the socket even. Attack surface would be incompariable to tor. :)
< wumpus>
the reason that Tor uses SSL, by the way, is that that masquerading as SSL is one of the few feasible ways to hide an encrypted connection in plain sight
< gmaxwell>
Indeed, for good reason. I don't think you're going to manage to hide that you're sending a transaction at all unless you use tor... but you well could hide which transaction you were sending. Esp if you don't mind broadcast taking a minute.
< wumpus>
if you start with your own such system you'd very soon run against the same issue
< gmaxwell>
(in particular it's completely reasonable to have high cover traffic, if you don't mind latency... just send a 100k message into the routing network every N minutes, containing some random assortment of transactions from the public network, and any of your own). If you're not running this over tor, you'll stand out as a user of the software but not more than that.
< gmaxwell>
)
< wumpus>
well you could still hide that you're sending a transaction by hiding in noise, e.g. also using the system to automatically send other people's transactions
< wumpus>
just like the relaying system we have now but encrypted, and high-latency, and with padding so that you can't look at the envelope and identify by the size, etc etc
< gmaxwell>
exactly. which is also good in that if someone finds out how to distinguish transactions exiting the system they might seek to punish their authors, but if you're sending random other transactions into the system, what exits will not just be from witting users.
< kanzure>
we are still missing out on a high-latency mixnet of some kind
< wumpus>
but anyhow - just like 5 years ago, I'd be pleasantly surprised if someone build something like that
< wumpus>
exactly kanzure
< gmaxwell>
prior to bitcoin they were hard(er) because of flooding attacks, but we have a one-two punch here: we have highly valuable tiny messages AND mechenisms to stop flooding attacks.
< wumpus>
lol "On January 23, 2006, the Russian FSB accused Britain of using wireless dead drops concealed inside hollowed-out rocks to collect espionage information from agents in Russia. According to the Russian authorities, the agent delivering information would approach the rock and transmit data wirelessly into it from a hand-held device, and later his British handlers would pick up the stored data by similar means." If
< wumpus>
everything fails, we could always use fake rocks :-)
< gmaxwell>
"some of us already are"
< randy-waterhouse>
yes, btcoin has been accused of using "fake rocks" many times already
< wumpus>
randy-waterhouse: any links?
< * phantomcircuit>
looks at scrollback
< randy-waterhouse>
wumpus: tongue-in-cheek reference to "fake gold" accusations
< MarcoFalke>
wumpus, using `-proxy` I get "2015-11-12 07:43:06 ERROR: Proxy error: connection not allowed
< MarcoFalke>
" for .onion and "2015-11-12 07:43:56 ERROR: Proxy error: host unreachable
< MarcoFalke>
" for the seeds
< MarcoFalke>
Are those implying censorship?
< wumpus>
randy-waterhouse: ahh old pyrite, or fool's gold :-)
< randy-waterhouse>
heh, right
< wumpus>
MarcoFalke: it may just mean what it says - the proxy cannot reach those hosts
< wumpus>
MarcoFalke: sometimes hosts are truly unreachable. I think you need to look at tor's logging to see if it can reach the first hop to troubleshoot this
< wumpus>
but at least you're one step further- you get errors from the proxy, instead of the level before of connecting to the proxy
< wumpus>
it may just have a hard time finding a working .onion peer here try this one: nns4r54x3lfbrkq5.onion
< MarcoFalke>
It only connects to nkf5e6b7pl4jfd4a.onion (TheBlueMatt)
< wumpus>
ok, in that case it's at least not censorship, if it can connect to one onion it can in principle connect to all without your ISP knowing anything
< MarcoFalke>
I don't have to set `-proxy=127...`?
< wumpus>
you do have to set a proxy
< wumpus>
otherwise it won't know how to connect to onions
< MarcoFalke>
Right now I am doing `-onlynet=onion -torpassword=`
< MarcoFalke>
and I can `-connect=` to it
< MarcoFalke>
but not sync
< wumpus>
it isn't able to derive the Tor proxy from the control port. Maybe this is possible, but it is not implemented.
< MarcoFalke>
I am doing `-connect=moheur3skn7jbca4.onion -proxy=127.0.0.1:9050 -testnet` locally and get the connection
< MarcoFalke>
But I need the proxy for outgoing connections, I see.
< wumpus>
ohh testnet
< wumpus>
probably very, very few onion testnet peers
< MarcoFalke>
Still, why wouldn't they sync each others blocks if connected?
< MarcoFalke>
Could you try `-connect=moheur3skn7jbca4.onion -proxy=127.0.0.1:9050 -testnet` on your machine?
< wumpus>
but this required P2P-level debugging, it's not tor's fault :)
< MarcoFalke>
I don't run master. Still running `f1d548d squashme: mention torcontrol in doc/tor.md
< MarcoFalke>
` with no merge
< MarcoFalke>
I guess thats an tested ACK on 6639, then?
< wumpus>
that doesn't matter - if it's the *other* peer that's ignoring you, nothing you can do
< MarcoFalke>
We are both in IBD?
< wumpus>
using debug=net I can see that I'm sending getheaders, but he's not responding
< wumpus>
or he's forked off the chain completely
< wumpus>
anything that would make the tip older than 24 hours old will preent him from sending headers
< wumpus>
I remember there were some issues on testnet
< MarcoFalke>
block 47762 is Jan 2013
< wumpus>
but he seems stuck there. blocks=... doesn't go up after reconnecting :)
< wumpus>
so we'll have to find a good testnet onion peer to check this
< MarcoFalke>
one sec...
< MarcoFalke>
going to sync and then `-connect=` again
< wumpus>
I can spin up one, after catching up (or I'll just comment out that check in #6971, so anyone-can-getheaders)
< wumpus>
ok one min
< wumpus>
MarcoFalke: tor: Got service ID vbzzvljohcyzlfzg, advertizing service vbzzvljohcyzlfzg.onion:8333 you should be able to sync from that
< wumpus>
eh wait that's not testnet
< wumpus>
it is: ytiwtue6no6c3cit.onion:18333
< wumpus>
may take some seconds for the new service id to propagate over the tor network
< MarcoFalke>
Could you make that `-connect=moheur3skn7jbca4.onion -proxy=127.0.0.1:9050 -testnet` and then sync me?
< MarcoFalke>
because I am still running with no `-proxy` to prevent outgoing connections
< wumpus>
ok
< wumpus>
I can connect, but I cannot sync from you, because your node is in IBD
< wumpus>
/Satoshi:0.11.99(hi_wumpus)/ hehe
< wumpus>
anyhow yes that's a tested ACK on #6639 - it's clear that creating the hidden service and makng it connectible works
< MarcoFalke>
Ok, then. Will do some more testing and comment on the pull. ;)
< phantomcircuit>
gmaxwell, so im (soft) disabling walletbroadcast but what am i doing about whitelisted peers
< phantomcircuit>
it seems that there's an argument to be made both ways
< gmaxwell>
phantomcircuit: relay for them; and add an option to control their relay special casing (e.g. so you can turn off the behavior that irritates me too. :) ) and then you can soft set that off too when blocksonly is on.
< sipa>
that seems neat
< phantomcircuit>
-relaywhitelisted ?
< gmaxwell>
whitelistalwaysrelay might be more descriptive.
< gmaxwell>
since turning it off should turn off the current behavior where we will accept txn from a whitelisted peer that we wouldn't otherwise.
< sipa>
right, normal relay for whitelisted peers would keep working even without the special case behaviour
< phantomcircuit>
default to true or false?
< phantomcircuit>
i guess default to true unless blocksonly=1
< gmaxwell>
default to true (current behavior) and soft-off it in blocksonly mode.
< gmaxwell>
then people who want to relay for nodes behind them while in blocksonly can turn it on.
< phantomcircuit>
gmaxwell, bleh it's now a hideously huge 13 loc patch
< phantomcircuit>
:P
< gmaxwell>
obviously you have to find some unrelated code to remove at the same time... (heh no not really)
< wumpus>
phantomcircuit: we don't make it easy to make a small patch, do we :)
< sipa>
we need patches that are C++-neutral
< phantomcircuit>
wumpus, lol @ constant
< phantomcircuit>
BUT BUT
< phantomcircuit>
wumpus, main.h or net.h?
< wumpus>
phantomcircuit: depends on where it's used
< wumpus>
I thought main.h
< sipa>
phantomcircuit: your patch will go into history books as the one with the higher discussion per lines of code patch ever that got merged
< sipa>
vRecvGetData has no business in CNode, for example
< wumpus>
right
< sipa>
how do i run the python tests from the command line?
< wumpus>
cd qa/pulltester; ./rpc-tests.py
< sipa>
ha, ok!
< wumpus>
and if you want to run a seperate test you can execute it manually from qa/rpc-tests
< sipa>
oh lol, i can't remember it being so easy
< wumpus>
whoa does anyone ever read README.md? :-) it still mentions the pull-tester
< GitHub93>
[bitcoin] sturles opened pull request #6997: Don't use hard-coded AllowFree, as this is usually far too low. (master...no-default-free-priority) https://github.com/bitcoin/bitcoin/pull/6997
< GitHub55>
[bitcoin] laanwj opened pull request #6998: doc: Remove mention of pulltester from README.md (master...2015_11_readme_pull_pulltester) https://github.com/bitcoin/bitcoin/pull/6998
< wumpus>
I also think the "Manual Quality Assurance (QA) Testing" is very much out of date
< GitHub86>
[bitcoin] jonasschnelli opened pull request #6999: add (max)uploadtarget infos to getnetinfos RPC help (master...2015/11/maxuploadtarget_rpc_help) https://github.com/bitcoin/bitcoin/pull/6999
< GitHub29>
[bitcoin] jonasschnelli opened pull request #7000: [Qt] add shortcurts for debug-/console-window (master...2015/11/qt_shortcuts) https://github.com/bitcoin/bitcoin/pull/7000
< jonasschnelli>
Yeah. PR #7000!
< phantomcircuit>
sipa, where else would vRecvGetData be?
< sipa>
phantomcircuit: CNodeState
< phantomcircuit>
ah
< phantomcircuit>
sipa, is the distinction you're thinking between the socket and the protocol things?
< phantomcircuit>
if so maybe CBitcoinNode : CNode
< sipa>
phantomcircuit: the idea (well, my idea... i don't know how many people share it) is that CNode should hold pointers to various module-specific state objects
< sipa>
phantomcircuit: and that when invoking code from one module, it just gets its own state objected handed to it
< sipa>
so ping handling can be entirely separate from block synchronization for example, and use completely independent locks, and even be totally unaware of the other's existance
< wumpus>
sounds like a sensible way to handle a protocol with different (mostly) independent concerns
< phantomcircuit>
yes except for the part where we decided to maintain the order of responses
< wumpus>
uhm, you mean that responses come in the same order as commands that trigger them?
< wumpus>
that's a pretty sane assumption that holds for most network protocols AFAIK
< sipa>
phantomcircuit: that's not incompatible
< sipa>
phantomcircuit: the ping module could be processing a message from node A, while the block sync logic is processing a message from node B
< phantomcircuit>
ah
< phantomcircuit>
wumpus, there's afaict no really good reason to enforce ordering of the responses actually
< phantomcircuit>
the only thing that relies on it in anyway is the initial block download thing
< phantomcircuit>
and im not sure it really does anymore
< sipa>
it does
< sipa>
normal block sync does too
< sipa>
bip37-based downloading does too
< wumpus>
phantomcircuit: I don't see a reason to make it different though
< sipa>
and a ton of test infrastructure relies on it
< wumpus>
is there anything specific that could be done better if it was not the case?
< phantomcircuit>
wumpus, yes we could improve the throughput of getdata requests if they were out of order
< phantomcircuit>
as it is there's a large latency hit if the block requested isn't in cache
< wumpus>
how?
< phantomcircuit>
wumpus, parallel fetch from disk
< wumpus>
getdata is already enough of a i/o DoS attack as it is, thank you
< sipa>
i don't think we should go write our own optimizer for disk fetches
< phantomcircuit>
hell we could even rewrite the requests so that they hit disk sequentially
< sipa>
phantomcircuit: you'd also need a reorganizer that rewrites the block files to be sequential
< sipa>
please don't go there
< wumpus>
TBH this doesn't sound very reasonable nor useful, please profile first before you think about optimizing anything
< phantomcircuit>
sipa, parallel fetch isn't anywhere near as complex as writing our own optimizer.... but it might need one if it's not to completely break hdds
< wumpus>
optimizing the disk component of getdata doesn't rank anywhere in the 'concerns that slow down bitcoin core in practice' I'm sure
< sipa>
phantomcircuit: so if that's really a concern at a time when the protocol too strongly relies on the ordering to meaningfully change it, we can always add a pargetdata which allows answers in any order. fixed.
< wumpus>
we already do parallel getdata on different nodes, that's even better paralleism thanks to sipa :)
< phantomcircuit>
wumpus, it basically only effects people like me syncing over gigabit lans from systems with the entire thing in page cache
< wumpus>
... right
< phantomcircuit>
increasing the fetch depth to 100 mostly fixes it
< morcos>
sipa: ha, i was chasing my own tail on that insecure_rand. Ok so should I fix the extra 30 bits of randomness or take our chances on only 32. I vote take our chances.
< sipa>
morcos: will the test fail if it actually is identical?
< morcos>
i was going to consider testing that. i was thinking that it might allow you to spend a duplicate coinbase which has already been spent
< morcos>
but maybe even thats ok, unless you randomly spend the second copy in exactly the same way. in which case it would break
< morcos>
i think?
< sipa>
morcos: hmm, your cache will be very slow
< sipa>
eh, small
< morcos>
it is slow, but what do you mean small?
< sipa>
eh, for some reason i was assuming that normal transactions decreased the number of entries... but they create a new output too of course
< morcos>
it about triples the time of that test from 300ms to 1sec
< sipa>
i guess you grow to around 4000 entries
< morcos>
yeah so on a typical test you generate around 4000 coinbases about 40 of which are duplicate and you end up towards the end of the test, failing on about 40 of them (if you have the broken fix)
< morcos>
that should be right
< morcos>
sorry around 400 of which are duplicate
< morcos>
b/c you only fail on the duplicate if it happens to get spent before its flushed. so i didn't check for coverage of that, since its a bit tricky, but that seemed a high enough failure rate that slight tweaks to the test shouldn't affect it.
< sipa>
you have around 1.7% chance to hit a duplicate during a test run :)
< sipa>
oh, wait, that's assuming there are only coinbases
< sipa>
around 0.2% chance per run; i think you need 64-bit amounts :)
< morcos>
how did you get 1.7%? does that sound really high? do you have to have the same random 32 bit number in 4000 tries (or 40k for the calculation you did)
< sipa>
i calculated 1 - multiply(1 - x/2^32, x=1..4000)
< sipa>
that's 0.2%
< morcos>
oh silly me
< morcos>
yes i calculated that wrong
< morcos>
ok i'll make it a 64-bit number with the cast.
< sipa>
of course, you could just use the iteration counter instead :)
< morcos>
good even easier
< morcos>
in a car now.. i'll push it in a bit
< morcos>
ok lots of traffic, squashed the fix in.
< instagibbs>
Hopefully not while driving :)
< morcos>
heh, no...
< jgarzik>
heh
< sipa>
Hopefully not while trafficking
< jgarzik>
with Uber & self driving cars, non stop hacking is possible :) Ah, the future.
< GitHub33>
[bitcoin] laanwj closed pull request #6639: net: Automatically create hidden service, listen on Tor (master...2015_08_tor_hs_v2) https://github.com/bitcoin/bitcoin/pull/6639
< GitHub51>
bitcoin/master 8f4e67f Wladimir J. van der Laan: net: Automatically create hidden service, listen on Tor...
< GitHub51>
bitcoin/master 2f796e5 Peter Todd: Better error message if Tor version too old
< GitHub51>
bitcoin/master 09c1ae1 Wladimir J. van der Laan: torcontrol improvements and fixes...
< jcorgan>
wow, i missed this PR entirely. what a great idea, glad it is merged.
< MarcoFalke>
yes, good to see this in 0.12!
< GitHub56>
[bitcoin] sdaftuar closed pull request #6992: [WIP] [Mempool] Add space for priority transactions (master...add-priority-to-mempool) https://github.com/bitcoin/bitcoin/pull/6992
< Luke-Jr>
morcos: ok, so back to your stuff? :P
< gmaxwell>
Luke-Jr: on priority. the exact current behavior requires walking all inputs for all transactioin in the mempool on every CNB to update priorities. With large mempools this is an enormous amount of time, and we cannot continue doing this.
< Luke-Jr>
morcos: can you make it so the accurate priority gets used with a policy option miners can enable? that makes current policy possible, right?
< gmaxwell>
There are multiple alternatives but most result in lots of software complexity, which we also probably don't want.
< morcos>
Hmm.. I think that would be possible
< Luke-Jr>
gmaxwell: it can be disabled by default
< morcos>
The original version of my pull did the expensive calculation
< morcos>
it'll make the code slightly uglier casing that out
< gmaxwell>
Luke-Jr: I think miners would be stupid to enable something that makes CNB 10x + slower, and we'd be stupid to offer it since slow CNB means more 'spv mining' and other bad effects.
< morcos>
but yes, i agree with that concern
< sdaftuar>
what's the motivation to keep the current version of priority rather than starting priority as the metric?
< Luke-Jr>
gmaxwell: CNB is not time-critical with Eloipool, and doesn't imply SPV mining
< gmaxwell>
I wouldn't _oppose_ an option, default off, except on the basis of carrying around code, and lack of testing (but I assume you'll test)
< morcos>
Luke-Jr: it may be something you can just easily patch in yourself
< petertodd>
sdaftuar: I think for most uses of priority, updating every time shouldn't make much of a difference - we're talking about old inputs after all
< petertodd>
sdaftuar: whats the difference betweek 1 month and 1 month + 1 hour?
< Luke-Jr>
I suggest we have it in Core for 0.12, and try to optimise (or remove) by 0.13
< gmaxwell>
sdaftuar: I think starting priority is a pretty poor approximation, though on the basis that I wouldn't be too opposed to ditching priority entirely, using a poor approximation isn't so bad.
< sdaftuar>
it just seems pretty arbitrary to me to decide to age inputs rather than use the age when relayed to you
< sdaftuar>
so given two arbitrary calculations, might as well do the faster one
< gmaxwell>
sdaftuar: age inputs has a property that the tx shouldn't hang around forever, but will actually bubble up and get mined at some point.
< Luke-Jr>
sdaftuar: well, aging inputs properly may in some cases be necessary for txns to confirm ever
< sdaftuar>
well we're about to deploy RBF and expiry
< petertodd>
the only time this matters is very high value outputs which we're expecting to accumulate priority quickly, and why incentivise people to put everything in on etxout? can have bad privacy implications
< Luke-Jr>
sdaftuar: as optional policies
< phantomcircuit>
Luke-Jr, then write a background thread to update priority
< petertodd>
phantomcircuit: +1
< morcos>
Luke-Jr: I'll see if I can whip up a dependent PR that adds back in the accurate calculation with an option, that could be included.
< Luke-Jr>
phantomcircuit: that would be one way to optimise it, but I hope to have something better for 0.13 anyway
< gmaxwell>
I do agree if the functionality was retained update in the background would be prudent.
< Luke-Jr>
morcos: well, I don't want to merge the existing PR without that included, so might as well put it in the same PR..
< morcos>
If you want to properly calculate priority, you need to do it dynamicalyl as in 6357 otherwise you destroy your UTXO cache
< morcos>
6357 is strictly better than that
< morcos>
Luke-Jr: yes but that may not be the prevailing view.
< morcos>
Well lets argue about it when we see what it looks like anyway
< phantomcircuit>
Luke-Jr, the priority calculation as it existed before mempool limiting cannot be reasonably calculated in realtime, the only way to do it is as a background thread
< wumpus>
* [new tag] v0.11.2 -> v0.11.2
< petertodd>
wumpus: +1
< morcos>
First I have to fix the existing pull, it uses the wrong calculation now anyway
< petertodd>
wumpus: or really, -rc
< Luke-Jr>
petertodd: you missed the rc
< morcos>
phantomcircuit: no, background thread is bad b/c of cache. see 6357
< petertodd>
Luke-Jr: lol, I mean, we're taking the rc off of v0.11.2rc1 :)
< Luke-Jr>
o
< phantomcircuit>
morcos, a background thread is equally as bad as any other way of doing it in that regard
< morcos>
6357 does not destroy your cache, and is way less computational complexity
< gmaxwell>
as far as this priority discussion, I think people are being far too dismissive about priority. In particular, this idea of fee competition vs dos attacks is a situation where the defender has no advantage on an attacker at all... they spend the same. Having a tie breaker that rewards coins that haven't recently moved is, I think, an elegent way to create a gap so that attackers don't merely nee
< gmaxwell>
d to pay market rate but slightly above. And impact to miner income can be made negligible.
< morcos>
gmaxwell: that argument doesn't make sense to me
< morcos>
i think the HOV/fast lane arguement makes sense
< morcos>
but to have priority boost fee doesn't make sense
< petertodd>
gmaxwell: why do we assume the defender always has more old coins than the attacker?
< morcos>
so it boosts it by delta. then attackers just have to pay market rate + delta, who cares, unless delta is large, in which case then you have prioirty txs crowding out fee paying txs and hurting miners profits
< gmaxwell>
morcos: right now the attacker need only bid episilon more than market rate to replace the honest users at market rate in the mempool.
< gmaxwell>
petertodd: it's a hurestic, and it maps to the actual attack pattern in that by their nature an attack requires making lots of transactions. Honest use may or may not have that property; but it remains a distinguisher, if not a super strong one.
< morcos>
its impossible to pick the conversion so that it accomplishes exactly your goal.. unless you reserve only a certain amount of space for priority, which is what the existing code does
< petertodd>
e.g. IIRC in lightning you can be in a situation where you need to get a recently mined tx spent, and the attacker may be trying to flood mempools to keep you from doing that
< morcos>
the existing code makes sense for that purpose, i agree. but combining to just one bucket filled on a combined metric does not
< Luke-Jr>
petertodd: I'm sure there are lightning-optimised policies that are possible?
< petertodd>
morcos: and *that* can be easily done with two separate mempools, each separately limited, which keeps the codebase simple
< petertodd>
Luke-Jr: better for privacy if you can't distingish lightning txs from othre txs
< Luke-Jr>
…
< gmaxwell>
morcos: Okay, you're right there. I was thinking that even having the attacker have to pay the extra delta is a benefit, but you're right that its probably negligible.
< Luke-Jr>
but you can
< phantomcircuit>
morcos, 6357 is less computationally complex, but significantly more complex code
< wumpus>
and another: * [new tag] v0.10.4 -> v0.10.4
< morcos>
phantomcircuit: ok that i will agree with!
< gmaxwell>
The two pools approach can also resolve the repricing cost issue, I think, if transactions cannot move back from the feerate pool to the priority pool, and if the priority pool is small.
< morcos>
sdaftuar's pull was 2 pools
< morcos>
maintained in the same structure
< phantomcircuit>
morcos, and it's a bunch of caching of values which is genuinely hard to get right
< gmaxwell>
(as you could limit repricing to the priority pool if the above assumptions are kept)
< gmaxwell>
maybe I just need to come around to bluematt's thinking that priority is not worth maintaining...
< dgenr8>
different-sized mempools mean that attacker doesn't have a perfectly defined function to erase a tx with a given fee
< morcos>
gmaxwell: i mostly agree, i wanted to keep priority, but it is a lot of complication, and maybe its better to realize that if we can come up with some incentive compatible metric like priority
< morcos>
then we can just add that back in in a clean way
< morcos>
we dont' have to permanently shut the door on a metric like this
< dgenr8>
also, attacker has to erase 299M of transactions to affect the lowest-feerate tx that's going into the next block
< dgenr8>
at default mempool setting
< sipa>
dgenr8: highest
< dgenr8>
300M to affect highest, no?
< gmaxwell>
morcos: yes, if block costing is changed, then it becomes incentive compatible and works better; though I am really really not fond of multi-metric block costs.
< morcos>
but block costs are inherently multi-metric with sig-op limit
< morcos>
you could buy extra block size with older age coins even
< phantomcircuit>
morcos, the sigop/byte limit is so high that no legitimate user is going to hit it
< sipa>
dgenr8: oh, i see what you're saying; in theory, yes, but in practice chains of transactions are harder to kick out and tend to linger, making it more spread out
< gmaxwell>
morcos: yes, and I want that to go away (it's busted anyways. :) ) fortunately the sigop limit is high enough that under ordinary use it's never hit.
< phantomcircuit>
you basically need to have a transaction that consists of OP_DUP OP_DUP OP_CHECKSIG repeated 100+ times to hit the limit
< morcos>
gmaxwell: ha, but you're the one tryign to worry about hitting it now b/c of the flood
< morcos>
if you dont' have multi metrics, then you have conversion factors between metrics which are hard to get right
< gmaxwell>
yea I know, I know, short term effects because of people specifically trying to exploit it.
< gmaxwell>
morcos: I don't know if for incentive reasons it matters if they're "right"
< gmaxwell>
in that I think mostly the sign matters. "oh I should prefer to make transactions that do X because it'll cost less"
< gmaxwell>
or at least only matters to within an order of magnitude.
< morcos>
gmaxwell: fair enough, but i'd say heuristics for optimizing multi metrics are also good enough for mining
< gmaxwell>
morcos: but will anyone ever implement them? Multiple limits also complicate the implementation of fraud proofs, making it less likely that we'll ever have them.
< morcos>
sure i will! but fraud proofs, hmm... because you ahve to be able to prove any one of those things?
< morcos>
i don't understand enough about how fraud proofs work
< gmaxwell>
yes, and efficiently proving a global limit requires commiting to a tree over its computation.
< * michagogo>
has his gitian-build script already running
< gmaxwell>
E.g. you commit to the value of of each input and output, and hashtree it, and in the interior nodes include the sum fees of the children. Now you can make an efficient proof that a block creates inflation. This can be done for any limit, e.g. sigops, whatever, but the interior nodes in the hashtree have to track each limit.
< gmaxwell>
morcos: I worry less that you won't implement it, and more that miners won't bother running code that almost never fires. "Morcos solved it" is a stable solution only if we assume people are running your code, which they might not for smarter or dumber reasons.
< gmaxwell>
morcos: also I think seperate limits are somewhat backwards in the sense that what we really seek to limit is "cost"; and turning cost into seperate limits is something we do because we don't know how to weigh for cost, but we do know what demand looks like, and demand uses a little of each dimension.
< gmaxwell>
in any case the issue with fraud proofs is mostly just making things even more complex reduces the odds that they get correctly implemented. Mempool hurestics are one thing-- they're not consensus critical, but limits are.
< wangchun>
Luke-Jr: We have priority size set to 0 for maybe one year. But I have blockminsize set to 250 KB.
< sipa>
wangchun: good to know!
< michagogo>
Also, I found out I'm a bit of an idiot. The issue I was having where as soon as I upgraded the LXC base, I had to keep it updated or the build would fail?
< Luke-Jr>
wangchun: let's chat in #bitcoin since there is conversation here already
< michagogo>
The fix is just to on-target -u root rm /var/cache/gitian/initial-upgrade
< michagogo>
(on the target, before copying it back over the base)
< michagogo>
(That's just a flag file that the upgrade script touches)
< morcos>
gmaxwell: ok, i'm not against weighted cost, i just worry about getting the weights right.. especially if those are consensus rules!
< jgarzik>
giving #6771 (chain limits) one last test run
< gmaxwell>
kanzure: that link is not correct in the details fwiw.
< michagogo>
Hm, weird lines from configure:
< kanzure>
quick overview of wrongness?
< gmaxwell>
(like saying utxo commitments are required, they're not)
< michagogo>
Checking for QT... no
< michagogo>
Checking for QT... yes
< kanzure>
okay, i'll keep that in mind.
< michagogo>
One right after the other
< sipa>
gmaxwell: until a few days ago i would have claimed that utxo commitments are required too :)
< gmaxwell>
sipa: pshaw, I've known they were for like ... months now. You need to spend more time in my head. :)
< sipa>
*walks back slowly*
< kanzure>
why few days ago?
< sipa>
kanzure: because that's when gmaxwell told me how to do it without commitments in a PM :)
< kanzure>
PMs don't count
< kanzure>
that's cheating
< gmaxwell>
it's been discussed in #bitcoin-wizards.
< sipa>
Oh.
< kanzure>
ah okay
< kanzure>
great
< gmaxwell>
in any case, you do them without a utxo commitment by instead commiting to the input block and offset that the input came from.
< gmaxwell>
then fraud can be proven by simply showing whatever is at that location, if its not the thing its supposted to be.
< gmaxwell>
I mean, I'm probably at fault for making people think that its the only way to get a fraud proof for spending non-existing coins, since in the very first writeup I suggest that https://en.bitcoin.it/wiki/User:Gmaxwell/features#Proofs
< gmaxwell>
but always things like that should be considered necessary, back then I didn't give it more than a moment's thought.
< gmaxwell>
er considered sufficient not necessary.
< kanzure>
ok, just posted follow-up to that thread linking to this explanation.
< gmaxwell>
(and obviously, spends of already spent coins proven by showing the other spend)
< phantomcircuit>
morcos, i dont really see the point of using a complex cost metric today as the principle limiting factor is bandwidth (and presumably will remain that way for many years)
< phantomcircuit>
if i was building a cost function is would be weighted to size so heavily as to be redundant
< jgarzik>
morcos, currently waiting on travis for #6771, then go for launch
< gmaxwell>
phantomcircuit: uh, even with libsecp256k1 dsa signature validation run at something like only 9mbit/sec/core (in terms of transaction size).
< morcos>
phantomcircuit: i thought you were arguing otherwise? but yes personally i think the biggest costs are bandwidth and utxo bloat. my idea was the metric should be something simple like .5 * total_bytes + 1 * output_bytes, but then maybe discounting provably unspendable output bytes
< morcos>
jgarzik: did it have a problem or you just rerun it for completeness
< jgarzik>
morcos, the latter - I always travis in jgarzik/bitcoin on the merged tree before pushing
< jgarzik>
travis results get stale rapidly
< gmaxwell>
phantomcircuit: and those figures are without multisig, it's worse with multisig. (like 2 of 3 is like 6mb/s/core)
< morcos>
gmaxwell: but you're saying that number is too low?
< morcos>
oh, i guess for a degenerate block that has all new sigs...
< morcos>
so maybe there is a standard cost metric, and then separate limits like sig op is now for the degenerate cases
< phantomcircuit>
gmaxwell, hmm that's slower than i was thinking
< phantomcircuit>
why was i thinking it was faster?
< * phantomcircuit>
doesn't know
< gmaxwell>
phantomcircuit: thats the only reason I mentioned it.
< gmaxwell>
Not that I thought it was super important here, but just because you were clearly thinking it was faster than that; you're probably thinking in terms of IBD with signature checking disabled.
< phantomcircuit>
gmaxwell, hmm maybe
< phantomcircuit>
gmaxwell, i know that we cant do >50mbps for that
< phantomcircuit>
possibly with some of the recent performance improvements we can improve on that significantly though
< morcos>
I think I have 2 backed up in moderation queue now
< michagogo>
Hm, I wonder if my script will break if there's an outstanding PR
< michagogo>
Actually, probably not. The Ruby oneliner that makes the PR is the last line in the script, so even if it fails and exits non-zero, the script is over anyway and there's nothing to abort
< petertodd>
gmaxwell: "utxo commitment by instead commiting to the input block and offset" <- that falls into the category of !@#$ why didn't I think of that :)
< michagogo>
(set -ex)
< CodeShark>
petertodd: that was actually my stumbling block to not fully getting some of the potential advantages of MMR trees
< CodeShark>
in retrospect, it makes a lot more sense to index outputs that way
< CodeShark>
at least for existence/nonexistence proofs - obviously for spending we need to reference the transaction in some way