< wumpus>
jonasschnelli: but for the 1MB buffer the benchmark will get called more times, also flattening out variance
< wumpus>
I also reasoned like you but forgot to take into account that the benchmarking framework automatically runs for a (approximate) fixed amount of time
< wumpus>
<Chris_Stewart_5> Is there a reason we don't have any logging inside of interpreter.cpp (or seemingly anything related to scripts)? <- for various reasons: it is inner-loop code, so logging would slow it down. Also it would be incredbily spammy. And further, any kind of logging complicates libconsensus initiatives by adding external dependencies
< GitHub135>
[bitcoin] jonasschnelli opened pull request #8046: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX (master...2016/05/fix_quit) https://github.com/bitcoin/bitcoin/pull/8046
< wumpus>
thanks for the benchmark results! and yeah re SHA extensions: I'm trying to target hardware that actually exist in the wild :)
< wumpus>
the AARCH64 sHA256 extension sounds reasonably interesting though, too bad Odroid C2 doesn't have it
< jonasschnelli>
Yes. Not really an important thing.
< jonasschnelli>
But enabling AVX should probably be something we should do.
< jonasschnelli>
wumpus: Is the implementation directly copied from intel?
< wumpus>
yes
< jonasschnelli>
1.5* performance boost in sha256 for advance vector enabled processors should be something we should try to get in soon.
< jonasschnelli>
I guess Sandy and Ivy supports it.
< wumpus>
what you could try is see if it influences sync performance (a matter of adding CSHA256::SetImplementation(CSHA256::Impl::SSE4); in the inititalization)
< gmaxwell>
s/AVX/SSE4/
< jonasschnelli>
SSE4 is ~the same performance boost on my Xeon. But right. We should probably go for AVX
< sipa>
building
< gmaxwell>
jonasschnelli: AVX is no faster on anything we've tested on and it hurts performance on Wumpus' AMD chip.
< jonasschnelli>
gmaxwell: arg. Right. Mixed it up.
< gmaxwell>
:)
< wumpus>
yes, AVX seems useless
< gmaxwell>
BlueMatt: can you benchmark this on skylake?
< jonasschnelli>
wumpus: so I just pass in a CSHA256::SetImplementation(CSHA256::Impl::SSE4) in init.cpp somewhere?
< wumpus>
jonasschnelli: yes, or alternatively change the default in src/crypto/sha256.cpp
< wumpus>
sipa: what processor?
< sipa>
Intel(R) Core(TM) i7-4800MQ, 2.6 GHz
< wumpus>
thanks
< gmaxwell>
sipa: thats another haswell.
< sipa>
ok
< midnightmagic>
I'm sure there's a reason for it, but is cpu detection and then dual codepaths, or even multiple-compile outputs not an option in the event a cpu family does benefit?
< midnightmagic>
or has a significant benefit to these features just not been found i guess
< jonasschnelli>
I could create a bench on a "2.6 GHz Intel Core i7" (guess this is Ivy Bridge) if it would be good to know.
< wumpus>
midnightmagic: runtime CPU detection is necessary to integrate this, yes
< wumpus>
(right now I don't bother as this is just a hack/experiment)
< gmaxwell>
midnightmagic: yes, it would be detected. But the test is important, e.g. it looks like we've learned not to use the AVX version even if supported.
< wumpus>
in any case it's nice that sse4 gives the best speedup; that's the most common
< * midnightmagic>
shakes fist at irritating icc dual code pathing which ((iirc) nullified it for commercial dev at prior employer
< gmaxwell>
well rorx was clearly faster in my benchmarks.
< wumpus>
but you actually have a cpu that supports it :)
< wumpus>
oh! you did post your results, I missed that
< gmaxwell>
I even highlit you. :)
< midnightmagic>
uh. I have a lemote laptop sitting next to me in a backpack. in the event someone might like me to use it for testing, I'd be cool with helping.
< midnightmagic>
(after I get it rebuilt)
< gmaxwell>
wumpus: so does sipa and jonasschnelli.
< gmaxwell>
They just didn't enable it.
< gmaxwell>
IIRC rorx was introduced with haswell so it's a couple arch generations old now and pretty widely spread. though I can't say if the modest performance increase is really worth it.
< GitHub57>
bitcoin/master 3262316 Chirag Davé: fReopenDebugLog and fRequestShutdown should be type sig_atomic_t...
< GitHub57>
bitcoin/master 2efe38b Wladimir J. van der Laan: Merge #8004: signal handling: fReopenDebugLog and fRequestShutdown should be type sig_atomic_t...
< GitHub29>
[bitcoin] laanwj closed pull request #8004: signal handling: fReopenDebugLog and fRequestShutdown should be type sig_atomic_t (master...fix_signal_handler) https://github.com/bitcoin/bitcoin/pull/8004
< jonasschnelli>
wumpus:do you have a tray icon menu on you Ubuntu?
< wumpus>
yes, but it's messed up (ends up in the lower left of my screen), a known (upstream) issue with qt5 and ubuntu 14.04, so I'm thankful for the option to disable it
< wumpus>
oh you mean if I can reproduce #8040 on ubuntu? I'll try
< wumpus>
no, I can't: the tray icon only appears after the main window shows
< wumpus>
it's not there during the splash screen
< wumpus>
this would be an issue if we had a "launcher icon menu"
< wumpus>
as the icon in the launcher appears as soon as the application loads
< wumpus>
the former doesn't seem to happen anymore though, so I think it was fixed upstream
< wumpus>
the latter only happens with self-compiled bitcoin-qt against the system qt5, whichi s ancient
< wumpus>
(5.2.1)
< wumpus>
in any case not worrying. The missing menu is strange
< wumpus>
adding the hide tray icon option is a good start, maybe at some point in the future it should be the default, too much trouble with it
< jonasschnelli>
Yes. That's what I was thinking. Though, the menu itself can be helpful. But the global menu space problem and the issues we always see with the tray menu make me think so.
< wumpus>
the right way to do this on ubuntu would be the 'launcher menu' route. But due to the practical concern with distributing GUI binaries on linux that'd only work, at most, for the ppa as that is custom-built for ubuntu
< wumpus>
in any case I'll try on ubuntu 16.04 too and see if I can reproduce 8043
< jonasschnelli>
wumpus: thanks.
< GitHub176>
[bitcoin] MarcoFalke opened pull request #8047: [qa] test_framework: Set wait-timeout for bitcoind procs (master...Mf1605-qaUtilTimeout) https://github.com/bitcoin/bitcoin/pull/8047
< GitHub130>
bitcoin/master 34ebceb Jonas Schnelli: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX
< GitHub130>
bitcoin/master 169d379 Jonas Schnelli: Merge #8046: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX...
< GitHub15>
[bitcoin] jonasschnelli closed pull request #8046: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX (master...2016/05/fix_quit) https://github.com/bitcoin/bitcoin/pull/8046
< GitHub194>
[bitcoin] laanwj opened pull request #8048: doc: Remove outdated qt4 install information from README.md (master...2016_05_doc_noqt4) https://github.com/bitcoin/bitcoin/pull/8048
< GitHub159>
[bitcoin] laanwj opened pull request #8049: Expose information on whether transaction relayed is enabled in `getnetwork` (master...2016_05_rpc_relaytxes) https://github.com/bitcoin/bitcoin/pull/8049
< GitHub116>
[bitcoin] laanwj closed pull request #7841: Expose information on whether transaction relayed is enabled in getnetwork (master...patch-2) https://github.com/bitcoin/bitcoin/pull/7841
< BlueMatt>
gmaxwell: still need it benchmarked?
< gmaxwell>
BlueMatt: we got skylake numbers from jonasschnelli
< BlueMatt>
kk
< jonasschnelli>
wumpus: benchmark results of full reindex up to height 400'000
< instagibbs>
I suppose I don't get the serialization code well enough
< jonasschnelli>
instagibbs: i think this line is not required. I guess I copied this from another class. Will have a look soon.
< instagibbs>
ok, that's my only ? left. Thanks
< luke-jr>
it's doing something there
< luke-jr>
nVersion is the method param, not the class var
< luke-jr>
I think READWRITE uses it?
< luke-jr>
definitely needs better docs
< instagibbs>
... what
< luke-jr>
+ inline void SerializationOp(Stream& s, Operation ser_action, int nType, int nVersion)
< instagibbs>
oh
< instagibbs>
right
< luke-jr>
I suggest renaming one of the nVersions
< luke-jr>
as-is, this is obfuscated code
< instagibbs>
agreed
< jonasschnelli>
Yes. Will rename it.
< jonasschnelli>
I think I took it 1:1 from CKeyMetadata
< wumpus>
jonasschnelli: nice, that's quite an improvement
< jonasschnelli>
wumpus: Yes. I guess most reindex work in ECDSA and not SHA. So yes. It's an impressive improvement.
< jonasschnelli>
I guess during a synced state the improvements are even more visible.
< jcorgan>
are there still thursday weekly dev meetings here?
< wumpus>
yes
< helo>
yes
< gmaxwell>
not for another 45 minutes.
< jcorgan>
ah, off by an hour
< droark>
Hi. Question about PR #7972. It adds a new file (qa/rpc-tests/create_cache.py) that creates a cache used, I assume, to assist with parallelizing the qa/rpc-tests code. I don't quite understand this. Can somebody help me out? TIA.
< jonasschnelli>
droark: You might need to ask MarcoFalke. But IIRC, you need to create the cache before you run the script in parallel because you run into the risk of different cache versions...
< sdaftuar>
jonasschnelli: droark: right, i assume it's to avoid having two jobs try to do it at the same time
< droark>
Sounds reasonable. What exactly is being cached, though? The results? The runtime environment for each instance? Something else?
< sdaftuar>
droark: it's a cached blockchain
< sdaftuar>
take a look at initialize_chain in qa/rpc_tests/test_framework/util.py
< droark>
Ahhh. Thanks! I'll ping Marco and double check the code but it sounds right.
< lclc>
Is anyone of the devs who are coming to Zurich arriving the day before (Thursday) ?
< sipa>
i'll be there on thursday evening
< jcorgan>
i get there thu evening as well
< lclc>
That doesn't surprise me :D I'm asking because I'd like to organize a meetup the evening before so the community also has something from the event :)
< jcorgan>
well, 6pm at the airport
< lclc>
Would be cool if someone would give a talk - topic is up to you. Our meetups are 30-110 people, free to attend, and recorded (www.bitcoinlectures.tv - YT Channel)
< wumpus>
I'm also arriving in zurich thursday evening
< sdaftuar>
meeting time?
< wumpus>
#meetingstart
< petertodd>
yup
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu May 12 19:00:44 2016 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< luke-jr>
on that note, it'd be nice if people didn't ACK/NACK bips they are not a listed author of ;)
< sipa>
was segwit testnet discussed on the ml?
< kanzure>
luke-jr: please elaborate.
< BlueMatt>
re: compact block, got good feedback from a few, made some slight tweaks but need to finish tweaks in the coming days, so no need to discuss here, I think?
< luke-jr>
kanzure: when it comes to a BIP draft, the only ACK/NACK that matters is the Author(s); others posting ACK/NACK just makes me spend time looking at the Author header to confirm I need to ignore it
< wumpus>
well in the case of #386 I felt I had the right to give my opinion because I was one of the people to come up with the idea last meeting, even though I'm not listed as author
< sipa>
i guess the question is where BIPs are to be discussed
< luke-jr>
sipa: bitcoin-dev ML
< sipa>
and the idea is that that would be the mailinglist
< sipa>
but...
< kanzure>
luke-jr: wouldn't NACKs from non-authors still be useful? why should the authors be the only trusted source of updates ? i don't understand.
< instagibbs>
oh, the compact blocks got a bip#
< paveljanik>
I don't understand either. If the author dies in accident, we can no longer change the BIP? or what?
< morcos>
luke-jr: i'd suggest that you just come up with a way to distinguish. for instance ACK/NACK shoudl be interpreted by the author and the author shoudl say Approved or Ready or something for your knowledge on when to merge
< kanzure>
luke-jr: i'm willing to believe you're right but you need better reasons yo.
< wumpus>
I agree that ack/nack isn't very useful, in comparison to more detailed comments
< petertodd>
kanzure: if I were author of a BIP an ACK would give me confidence; a NACK could be useful criticism
< sipa>
paveljanik: an active BIP can't be modified anyway
< kanzure>
wumpus: right, sure.
< luke-jr>
kanzure: the BIP is a document by the Author(s). ideally, they should have exclusive merge access for changes, but GitHub doesn't support that
< paveljanik>
text clarification, etc...
< kanzure>
okay this is just trivial though, luke-jr isn't going to stop me from ACKing on BIPs i didn't author :)
< morcos>
sipa: in some cases it should be, if for instance the BIP wording is incorrect ala 34
< luke-jr>
paveljanik: BIP 1 technically allows me to reassign BIP Author in some cases IIRC
< sipa>
morcos: yeah - makes sense
< jonasschnelli>
We should also define a rule how we deal with links to implementations that have implemented the BIP.
< sipa>
jonasschnelli: yes
< jonasschnelli>
IMO we should not add any implementation links.
< jonasschnelli>
(expect of sample impl.)
< wumpus>
many bips have a list of implementations; what's wrong with that?
< jonasschnelli>
Its just noisy
< sipa>
wumpus: in BIP32 there are continuously pull requests to add links
< gmaxwell>
it gets flooded with spammy updates.
< wumpus>
I think it can be useful to have e.g. implementations in many languages
< jonasschnelli>
And we don't control the implementations I guess
< sipa>
wumpus: which imho function as nothing more than advertizement
< kanzure>
and then we have to moderate the additions
< jonasschnelli>
sipa: Yes!
< gmaxwell>
And if we're not looking at it, we're eventually going to get a malware example BIP32 impl. :)
< wumpus>
ok...
< luke-jr>
BIP 2 comments would be a nice place to list implementations, but that was controverisal
< jonasschnelli>
I could imaging more and more advertising like PR to come up in future.
< wumpus>
I think the requirement should at least be to point to the code
< wumpus>
not just the product
< paveljanik>
so what is the topic now? ;-)
< wumpus>
to prevent advertizement
< kanzure>
i think the topic was "feedback about compact block relay BIP things"
< jonasschnelli>
Which still intersects (a little bit)
< jcorgan>
it is useful, however, when reading a BIP, to at least get pointed to reference code, but it need not be every implementation that anyone wants to list thereafter
< wumpus>
I think the topic is how to handle BIP implementations, which is as good a topic as anything
< wumpus>
but then you get the fight about what is reference code and what is not
< jonasschnelli>
Links are always difficult. You need to check them... then what if a link diverges from the BIP specification?
< jonasschnelli>
Do we check it?
< wumpus>
in some cases it's clear if the BIP author wrote code themselves
< kanzure>
yeah another threat vector is someone selling their github account in the future
< kanzure>
and then a BIP links to someone's github
< jonasschnelli>
Yes. These are the link we should keep there...
< jcorgan>
or the BIP author themselves pick one or more...
< gmaxwell>
If it's upfront then we can trust the bip author and review process to have had some chance to verify it.
< wumpus>
yes, I'd say it's up to the BIP author
< wumpus>
like other changes to the BIP
< kanzure>
other problem is keeping track of upstream updates like security fixes, oops. anyway, this is a lot of work.
< kanzure>
we should include hashes of the reference source code, in the BIP text
< luke-jr>
maybe BIP PRs should be PGP signed
< wumpus>
this is not something that should be globally decided
< gmaxwell>
I could see the criteria being different for different BIPs.
< jonasschnelli>
wumpus: Yes. This makes sense.
< kanzure>
by hashing the source code we can at least have a way for readers to verify that the source code is still the part we meant to hyperlink to :)
< wumpus>
gmaxwell: exactly
< wumpus>
I mean the author of bip32 could say 'enough is enough'
< jcorgan>
er, link to a URL and commit hash?
< wumpus>
some other BIPs have far fewer implementations and the author may be happy to see one
< gmaxwell>
(unfortunately, it's BIP32 that gets 95% of this-- key generation is an especially awkward place for random found on the internet code)
< petertodd>
my BIP65 links to two separate demo repos that just give some sample code, which is probably something we can generally consider as acceptable (ignoring the issue of more complex implementations that aren't pure demos)
< jonasschnelli>
I like jcorgan idea to link to sourcecode-only with a commit hash
< jonasschnelli>
But kinda static.
< wumpus>
Itend to agree with that. Link to the actual code, not the product
< wumpus>
not 'blawallet implements bip32', no, *the code linked here shows how we implemented BIP32 in language Z*
< sipa>
wumpus: agree
< luke-jr>
+1
< petertodd>
wumpus: +1
< paveljanik>
yes
< jonasschnelli>
ack
< wumpus>
ok
< kanzure>
we should review existing bips for source code links that do not include a commit hash. branch names are not OK.
< kanzure>
i mean, branch names without a commit hash are not OK.
< wumpus>
other topics?
< paveljanik>
Revies Jonas' HD - #8035
< jonasschnelli>
I have a tiny topic proposal that is very into the impl. teretorry..
< jonasschnelli>
RPC long poll notifications
< kanzure>
have we received an overview from sipa yet about areas of segwit that he feels should be most thoroughly reviewed
< sipa>
kanzure: no, sorry
< kanzure>
can we get 10 volunteers to heckle sipa about this?
< jonasschnelli>
RPC long poll would would allow remote GUI and remote wallet with a very easy setup. IMO it could allow third party developers to write simple remote GUIS, web-frontends, etc.
< jonasschnelli>
I would even say RPC long poll has more potential then ZMQ for core.
< wumpus>
I'm kind of divided on the notification thing to be honest - I'd prefer to stick to only one notification mechanism in bitcoin core (apart from the silly -Xnotify), and somehow zmq got a precedent there
< luke-jr>
calling pollnotifications myself seems like it would disrupt an application relying on it?
< jonasschnelli>
I also don't like multiple notification channels.
< luke-jr>
ie, we need channels
< jonasschnelli>
But how would you connect a remote GUI over the internet...
< luke-jr>
wumpus: zmq is crazy though :<
< luke-jr>
also note we already have longpolling
< wumpus>
yes it should definitely have multiple channels, the current code supports only one client
< jonasschnelli>
wumpus: no
< wumpus>
luke-jr: where were you to NACK zmq when it was added?
< jcorgan>
i'm happy to look at improving zmq
< luke-jr>
jonasschnelli: no RPC over internet ever :<
< jonasschnelli>
I have added support for <n> clients
< luke-jr>
wumpus: just because zmq is crazy doesn't mean optional ZMQ support should be excluded ..
< jonasschnelli>
RPC over internet over a reverse proxy is a wide use pratice.
< wumpus>
(I don't think zmq is necessarily crazy)
< jtimon>
sorry, what are the complaints with zmq? it is optional anyway
< wumpus>
jtimon: I'm not sure either, it seems to be fashionable to complain about it
< luke-jr>
jtimon: it's not optional if it is an excuse to exclude other systems
< luke-jr>
zmq breaks protocol compatibility for minor bumps
< jonasschnelli>
Working towards decoupling of the GUI and the wallet requires well defined channels/APIs
< luke-jr>
ie, 4.1 won't work with 4.0 right
< jonasschnelli>
ZMQ would require a tunnel (VPN, stunnel, etc.).
< luke-jr>
also see all the Elements functionary problems due to ZMQ
< jcorgan>
ZMQ 4.x is implementing CurveCP
< jtimon>
I think it's a wonderful tool I would like to use more (although maybe the fact that its creator rewrote it completely in nanomsg may indeed indicate that some parts of zmq have become too complex)
< wumpus>
zmq is basically only useful for local system, but so is RPC, it's not meant to be used over the internet
< jcorgan>
see comment about curvecp
< wumpus>
if you write tunneling for RPC - why not tunnel the notifications too?
< jonasschnelli>
I just think Core users would love to have a GUI/wallet-frontend that can run on a different machine.
< jtimon>
wumpus: exactly, it's for your intranet or at most inside a vpn (although I haven't tried that myself)
< luke-jr>
jonasschnelli: yes, for years now that has been desirable
< wumpus>
jonasschnelli: I agree, but does that need RPC notifications?
< jonasschnelli>
wumpus: RPC does work extremly well in reverse-proxy mode.
< wumpus>
jonasschnelli: what would you use it for?
< luke-jr>
wumpus: right now it requires polling
< wumpus>
e.g. to get notification of transactions/blocks the P2P protocol works fine
< jonasschnelli>
Look at rtorrent or any other RPC daemon software.
< jonasschnelli>
Polling is extremely inefficient. Long polling would allow clients to get "realtime" events while not require any other dependencies...
< jonasschnelli>
And the code changes are tiny...
< wumpus>
yes, agreed jonasschnelli
< wumpus>
yes the code changes are tiny
< jonasschnelli>
You could setup a save and secure remote bitcoind with a single apache/nginx config.
< wumpus>
but I'm a bit afraid the same will happen as with REST, people will want to pile up things on top, and now for every notification people will want zmq and rpc longpull support
< jonasschnelli>
Now do the same with ZMQ notifications... :)
< wumpus>
just like people want every possible data item both through RPC and REST
< sipa>
well ZMQ/P2P are suboptimal to write a remote GUI, as you can't filter for just wallet transactions
< wumpus>
I believe thatthere is value to keeping core limited to one interface
< luke-jr>
wumpus: I don't see why this is a problem
< jonasschnelli>
Yes. If i had a blank paper. I would drop ZMQ and REST in favor or RPC longpoll and the normal RPC.
< jcorgan>
longpoll does indeed solve some of my original motivation for adding ZMQ
< sipa>
jonasschnelli: i think you're biased because you're only thinking about one use case
< wumpus>
well my iniitial idea for notifications was also something like longpoll, or a streaming socket
< wumpus>
then a proxy from that to zmq
< wumpus>
but zmq was first
< luke-jr>
zmq is optional
< luke-jr>
someone shouldn't need to install zmq for notifications
< jtimon>
zmq's req/rep could replace both rpc and rest
< jonasschnelli>
There is one big advantage of RPC long polling...
< jonasschnelli>
you can have private notifications...
< jonasschnelli>
secured behind auth
< wumpus>
do we want private notifications?
< jonasschnelli>
Like a new relevant wallet tx comes in
< sipa>
wumpus: for a remote wallet gui you do
< jonasschnelli>
It would be on the same level then the RPC security...
< jonasschnelli>
just instead of poll you can have push
< wumpus>
connect a remote wallet GUI through RPC?
< wumpus>
wasn't RPC meant to be for local usage?
< jonasschnelli>
I would not add this over ZMQ because of the unknown risks
< wumpus>
I think the idea was to attach a *wallet*, not a wallet GUI
< wumpus>
the wallet needs to be split from the core
< luke-jr>
a wallet can be attached over p2p fine
< jonasschnelli>
Another solution would be to provide a tiny daemon that would interact between core <-> remote GUI/wallet.
< wumpus>
I'd prefer for core to handle semi-public data, then have a more restricted wallet
< jonasschnelli>
But that tiny daemon would speak RPC with the outside/internet.
< luke-jr>
ideally we should have node <-> wallet <-> GUI
< wumpus>
luke-jr: yes
< jonasschnelli>
luke-jr: But how would you see the communication channel between wallet <> GUI?
< jcorgan>
another motivation for adding ZMQ was to allow bitcoind to be a trusted "border gateway", that spoke P2P and consensus, and then stuff behind it would be very simple ZMQ-based software that didn't need to know all about those things
< wumpus>
but wallet<->GUI could be a competely different protocol than node<->wallet
< jonasschnelli>
Needs to be bidirect.
< luke-jr>
jonasschnelli: see wiki link; or something like what you're doing
< jonasschnelli>
node <> wallet is probably p2p?
< sipa>
yes
< luke-jr>
yes
< morcos>
node <-> wallet being p2p leaves a lot of missing pieces
< wumpus>
maybe authentiated P2P
< morcos>
fee estimation
< jonasschnelli>
But what direction do we want to go for <wallet> <-> <gui>?
< wumpus>
as you proposed
< wumpus>
with some private extensions
< sipa>
jonasschnelli: up to the wallet
< morcos>
mempool actions (eviction, how close to top of mempool, whether it was accepted)
< jonasschnelli>
morcos: fee estimations can be done with authentication/private extensions.
< BlueMatt>
ehh, lets not layer everything onto p2p extensions
< luke-jr>
jonasschnelli: XMPP! /s
< jtimon>
why people want to remove zmq? I still don't undesrtand
< jonasschnelli>
sipa: That's why I propose RPC long poll (as a "wallet" feature). :)
< sipa>
jonasschnelli: i'm not sure our wallet should provide that
< sipa>
jonasschnelli: at least not at this stage
< wumpus>
BlueMatt: well if we have a autenticated+encryped P2P protocol, adding private extensions is attractive
< wumpus>
jtimon: I don't want to remove zmq. But I do think we should have one notification mechanism.
< kanzure>
notifications over zeromq would be nice
< jtimon>
wumpus: why not it be zmq?
< luke-jr>
jtimon: I just want to keep ZMQ as an optional feature, not necessary for this stuff
< wumpus>
jtimon: I don't want to amintain an endless pile-up of different external notification mechanisms
< jonasschnelli>
Yes. We don't want that.
< jtimon>
wumpus: ok, why not use ONLY zmq?
< BlueMatt>
wumpus: its quite attractive to shove everything into one monolithic protocol, indeed, but I do think there is a lot of value in splitting things out (though I wouldnt be upset if they were logically different code that just happened to have the same on-wire protocol or so)
< jcorgan>
jtimon: i agree that zmq req/rep and pub/sub could provide the entirety of needed interfaces to bitcoind, but that's an argument lost years ago :)
< luke-jr>
jtimon: then it's no longer optional
< kanzure>
unfortunately the simplest private notification stuff that the rest of the community would understand would be.... web hooks. :(
< luke-jr>
jcorgan: except ZMQ has no protocol compatibility
< wumpus>
websocket would work
< morcos>
wumpus: i think we can move towards deprecating some things that we view as redundant. what we should do now though is decide what would work for a node <-> wallet communication protocol. b/c that is something we defintiely want.
< jtimon>
luke-jr: ok, either we have 1, we remove zmq or we make it non-optional
< luke-jr>
wumpus: websocket doesn't define a protocol
< wumpus>
but just as well as longpoll
< jonasschnelli>
websockets have bad security
< sipa>
wumpus: i think it may be reasonable to say that zmq is for node notifications (which are unauthenticated) and longpoll for wallet notifications
< luke-jr>
jtimon: if we can only have 1, then IMO zmq can go
< wumpus>
morcos: what would you propose to deprecate?
< jtimon>
luke-jr: what do you mean by "protocol compatibility"?
< luke-jr>
jtimon: but I don't think we should limit to 1
< wumpus>
sipa: but what jonasschnelli wants is not wallet notifications
< luke-jr>
jtimon: ZMQ 4.0 can't talk to ZMQ 4.1
< jtimon>
luke-jr: and IMO, if we only want to have one, zmq should stay
< sipa>
wumpus: yes it is
< wumpus>
sipa: he wants the same stuff as is now offered over zmq
< jonasschnelli>
wumpus sipa: What i want is going toward wallet notifications
< morcos>
wumpus: well my argument is to first define the right way to do node <-> wallet and then do that (even if it means adding something) and then we can revisit and see what we have that is extraneous
< jonasschnelli>
But also notifications that could enable a remote GUI
< sipa>
i don't think so; nothing of what is offered over ZMQ is what you need for a remote wallet GUI
< jonasschnelli>
(mempool / peers, etc9
< jonasschnelli>
A Core node GUI on a smartphone....
< jonasschnelli>
Which can't work over ZMQ
< sipa>
i think we shouldn't discuss that in the current stage
< jcorgan>
of course it *could*
< jcorgan>
but not as it is written today
< wumpus>
jonasschnelli: how would you protect the RPC connection to the smartphone?
< sipa>
i don't want my bitcoin core full node software to be accessible by smartphones on the internet, i think
< kanzure>
jcorgan: how much difference?
< wumpus>
jonasschnelli: the same tunneling tool could route the notifications from zmq
< sipa>
except the p2p protocol, which is must provide by necessarily
< morcos>
jonasschnelli: don't you think we shoudl concentrate on node <-> wallet now? b/c then different people could build different wallet <-> gui protocols if we wanted. our core responsiblity is the node
< jcorgan>
you have to define a protocol/serialization/encapsulation to run over it
< luke-jr>
jcorgan: except then you need to make sure your ZMQ lib versions match, which is just annoying
< jcorgan>
you can say that about dozens of other libs
< jonasschnelli>
morcos: I'm working on node <-> wallet. That's why I'm working on auth/enc. :)
< jtimon>
jonasschnelli: how can something "not work over ZMQ"? can't you proxy the messages through some other protocol outside of bitcoind once you get the zmq messages?
< luke-jr>
jcorgan: most protocols are compatible across lib versions
< sipa>
jtimon: because authentication
< jonasschnelli>
I don't want to talk SPV over unencrypted channels...
< kanzure>
unencrypted...?
< jcorgan>
anyway, if anyone wants to pursue making changes to the current zmq implementation, we can talk in zurich about it
< wumpus>
wellt he same protocol that offers RPC over the internet needs authentication too
< jonasschnelli>
jtimon: Sure can you. But can a normal user do that?
< wumpus>
you have exactly the same problem there
< jonasschnelli>
I just compared the requirements to setup a remote rtorrent GUI with a possible remote Core node GUI.
< wumpus>
in any case, I'm not strongly against longpoll RPC notifications
< jtimon>
jonasschnelli: well, what was your "normal user" using isntead of zmq?
< jonasschnelli>
And i feal that rpc long polls would result in a easy setup...
< sipa>
morcos, wumpus, jonasschnelli: i agree node <-> wallet is what we should be talking about first, and the apparent need to add private extensions is a sign of a deeper problem: an SPV wallet without trusted full node will be lacking in features
< kanzure>
easiest setup for non-core-developers is going to be web hooks :(
< kanzure>
not rpc
< d4de>
0MQ supports GSSAPI and Curve auth, unencrypted?!
< wumpus>
and indeed, RPC longpoll can be supported without linking against external dependencies
< morcos>
sipa: good point, but maybe thats a separate problem to solve?
< wumpus>
which is an advantage
< jonasschnelli>
</rpc-long-poll-advertizing> I'm happy to keep the PR alive... I'll also try to work on a intermediate script that could -> ZMQ and does -> RPC
< sipa>
morcos: well if that problem is solved (unsure how...), maybe we don't need private extensions :)
< sipa>
jonasschnelli: you can't... ZMQ doesn't offer wallet-filtered events
< morcos>
yes, but it'll always be superior to some degree to have access to a trusted node
< instagibbs>
sipa, do you mean things like access to mempool, etc?
< wumpus>
at the least we should make a clear division about *what* should be offered over RPC longpoll and what over zmq and what over (theoretic) P2P extensions
< sipa>
instagibbs: yes, and fee estimation
< morcos>
so we shouldn't limit ourselves to things that you can't do without that
< jonasschnelli>
sipa: you still can talk RPC...
< wumpus>
so not everything ends up on all three
< jonasschnelli>
sipa: just the notifications need to be ZMQ to avoid polling..
< instagibbs>
even if you have access to full node over p2p, that doesn't get you that (maybe that's what you meant)
< jonasschnelli>
sipa: thats why I think adding long poll would not change the security aspect.
< sipa>
jonasschnelli: but you don't want to send a ZMQ event for every transaction over the wire... that would consume as much bandwidth as just the p2p protocol
< wumpus>
adding longpoll to RPC won't change any security aspect
< wumpus>
(except if it encourages opening up your RPC port to the internet)
< jonasschnelli>
sipa: Right. Long poll could then be extended to relevant wallet txes.
< wumpus>
kanzure: you need user/pass for that, so only the owner can attack it
< wumpus>
kanzure: I would be against unauthenticated longpolls, that's for sure
< luke-jr>
sipa: at least the main protocol design (it transmits binary data as-is, IIRC)
< wumpus>
kanzure: and if you can authenticate you can do *much* worse things than hold up threads
< jtimon>
sipa: of course you can filter stuff with zmq, you can do anything with zmq, is "network complete"
< jtimon>
you may need more processes
< sipa>
jtimon: that's like saying that x86 asm is better than http, because it can do everything
< jonasschnelli>
Another + for RPC long poll: no dependencies...
< sipa>
jtimon: of course you can filter, but the filter would be too late
< wumpus>
so anyhow: I'd say continue your work jonasschnelli
< jtimon>
sipa: I really don't undesrtand what you claim can't be done with zmq
< kanzure>
my zeromq person just left the channel in a huff ("these people are retarded") hah...
< jonasschnelli>
wumpus: Okay.. I keep the PR warm.
< luke-jr>
again, we ALREADY have longpolling, so I don't see why make a big deal of it
< jtimon>
sipa: you want to filter by a set of addresses or something?
< jonasschnelli>
luke-jr: Yes. It's not a big deal...
< sipa>
jtimon: we don't want ZMQ to be authenticated, so it can't provide wallet-specific information, which means the ZMQ client will need to do the filtering
< wumpus>
people seem to have trouble to get zmq to work in practice, maybe if you can provide some easy examples for RPC longpolling then it will be the more popular way to do notifications soon
< jonasschnelli>
sipa: Agree
< sipa>
jtimon: which is duplicating logic, because that's something the wallet should do, not the gui
< jonasschnelli>
wumpus: Okay. I'll add some examples... good point!
< wumpus>
(I had so much trouble convincing joinmarket to use zmq notifications instead of wacky -notifyX scripts that I just gave u)
< luke-jr>
LP is really simple. just make a normal request and wait ☺
< jonasschnelli>
wumpus: hah .. yes. Another +1 for RPC long poll. :)
< wumpus>
maybe it's the extra dependency, maybe it's unfamiliarity, I don't know.
< jonasschnelli>
People will call exes on every transaction...
< jtimon>
sipa: oh, you don't want authenticated connections going through zmq processes...I don't undesrtand why, but ok, that seems to be the piece I was missing
< sipa>
jtimon: so yes, of course, you can do anything, if you reimplement the wallet in the client, but that was exactly what we were trying to avoid :)
< gmaxwell>
luke-jr: lots of things time out.
< jonasschnelli>
gmaxwell: time out doesnt matter
< gmaxwell>
this discussion seems ratholed.
< jonasschnelli>
Because you have a queue/state on the server
< wumpus>
timeout doesn't matter with longpoll, you can just re-request
< jtimon>
sipa: no, I would just use some kind of authentication
< kanzure>
we need more user metric survey data about whehter rpc is really easy for folks. i think all the web app developers only know web hooks and HTTP.
< wumpus>
kanzure: moving away from JSON-RPC as the main RPC API is out of the question at this stage
< wumpus>
just too much software and libraries that assume it
< sipa>
are there other topics?
< kanzure>
er i did not mean to unintentionally advocate for moving away from json-rpc
< wumpus>
doesn't seem to be
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu May 12 19:51:11 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< jcorgan>
do we have any grand plans or even wishlists for zurich?
< jtimon>
kanzure: I did suggest replacing json-rpc with zmq json req/rep, but I knew very few people would consider it seriously
< sipa>
i'd like to talk about p2p encryption, segwit, signature aggregation, ...
< jonasschnelli>
Agree with sipa. We could also form groups to directly work on stuff.
< sipa>
but the overall goal should just be to facillitate improvements that are in the pipeline by meeting in face... whatever those improvements are
< kanzure>
i would like to do braindumps to get things written down
< jonasschnelli>
+1 for writing down things...
< kanzure>
i am good at this
< jcorgan>
suggest doing as much brainstorming/dumping in advance to not use precious F2F time
< kanzure>
would like to draw up list of too-often-unspoken proposals to write down more details about
< sipa>
that'd be great
< jonasschnelli>
I think we should't feel too much pressure. The fact that we are meeting face to face alone is a value.
< jcorgan>
of course
< sipa>
agree
< jonasschnelli>
Even if there is no outcome. It would be a success.
< kanzure>
some feedback from d4de- he recommends that future bitcoin core meetings shoud have someone that more actively guides discussion, e.g. make things more explicit rather than implicit
< wumpus>
jtimon: right - there's no deep reason why zmq req/rep wouldn't work for the main RPC API, but it's .... unfamiliar for people, see my comment above about trying to get people to even use zmq for basic notifications
< jcorgan>
the face time is extremely valuable, just want to make the most of it
< kanzure>
jonasschnelli: gosh that sounds like one of those unreasonable metrics :) (kidding)
< jonasschnelli>
kanzure: It's called positiv thinking. :)
< wumpus>
kanzure: are you volunteering?
< jcorgan>
so if there are things that can be done ahead of time that don't necessarily need the f2f time, it would make sense to get it out of the way beforehand
< kanzure>
wumpus: for writing things down? yes. for brainstorming? sure... for meetings, i'll defer to our friendly neighborhod pointy-hair.
< jonasschnelli>
kanzure: I think wumpus is doing a fantastic job and you easly overmoderate on IRC.
< gmaxwell>
wumpus: execpt the no stable on the wire format, and no reliable delivery...
< jonasschnelli>
jcorgan: Agree. You could start a discussion on the bitcoin-core-dev ML?
< kanzure>
jonasschnelli: i don't have enough context on many of these topics to correctly guide meetings. so it's not something you should be concerned about. heh.
< wumpus>
gmaxwell: I think you can have reliable delivry over zmq, just not in the way we use it
< jtimon>
wumpus: yeah, it seems people forgot about socket-like things
< wumpus>
jtimon: well it's a complete new API to learn
< wumpus>
jtimon: even if you already know normal socket programming, zmq is like a whole new alien landscape
< jonasschnelli>
jtimon: RPC long poll require no "real" dependencies, you just aim a http request and wait until you get response.
< jcorgan>
wumpus: jtimon: people don't seem to grok message based vs. function call based
< kanzure>
jonasschnelli: that can cause an unintentional DoS against yourself
< jonasschnelli>
kanzure: explain?
< kanzure>
rpc thread exhaustion, as mentioned above
< jtimon>
wumpus: I hadn't used sockets in ages and it was very easy to me (but as said I just use xreq/xrep, push/pull and pub/sub, no dealers or other stuff)
< jcorgan>
jonasschnelli: i'll make a suggestion on the ML then
< jonasschnelli>
jcorgan:+1
< jonasschnelli>
kanzure: no.. you only run the poll http request on a single thread...
< jonasschnelli>
Only one request in parallel.
< kanzure>
jonasschnelli: rpcthreads
< jonasschnelli>
But thats the serverside.
< kanzure>
yes.. that's still DoSing yourself.
< kanzure>
you're not enforcing one request in parallel
< jtimon>
wumpus: but apparently nanomsg simplifies things further removing the context and things like that
< wumpus>
in principle you don't even need a rpc thread open for a long-poller, if you'd use the event based http, but that's a topic for the far future
< jonasschnelli>
kanzure: sure. But that would not be different to the "normal" rpc calls?
< kanzure>
jtimon: in addition to nanomsg, look at nq
< kanzure>
jonasschnelli: well, not many of them are meant to be open for a while
< jtimon>
kanzure: nq? do you have a link?
< jonasschnelli>
Sure. But you control them. Its secured behind auth.
< jonasschnelli>
kanzure: No strangers will connect to your RPC.
< kanzure>
jtimon: actually no :) i think this means i owe you BTC... hah.
< wumpus>
and if you need multiple long-pollers you can just increase the number of rpc threads anyhow, it's not like we need to support 1000 people in a chat
< wumpus>
no need to overdesign/overcomplicate
< jonasschnelli>
Indeed. :)
< jtimon>
jonasschnelli: you can have an http server with auth that connects "locally" (intranet) to stuff via zmq
< kanzure>
manual thread management, got it..
< luke-jr>
we're moving the dev meetings to the p2p network and using longpolling with bitcoin-cli to read the chat? :P
< jtimon>
jonasschnelli: but yeah, I guess that's one extra step you would be saving
< jonasschnelli>
jtimon: Yes. Sure. But why the roundtrip when you can have RPC long poll?
< kanzure>
we don't mention in our docs "don't connect to your bitcoin node from a thousand workers that listen for notifications"
< jonasschnelli>
jtimon: you don't need the ZMQ dependencies if you want to serve a push not. over http
< wumpus>
kanzure: some things are common sense, but feel free to add it explicitly
< wumpus>
kanzure: and for that, zmq is indeed more suited as it has one-to-many broadcast
< jtimon>
jonasschnelli: of course, strictly speaking "you don't need anything" to do anything, it's the "you can't do X using zmq" that nerves me
< jonasschnelli>
I think ZMQ is useful....
< gmaxwell>
jtimon: being excessively pedantic is not productive.
< jonasschnelli>
Its just pain in the ass to get a notification to a remote end.
< jtimon>
gmaxwell: sure, I was just misunderstanding some of the "you can't do this with zmq" claims, they were really just "you also need X" or similar things
< jtimon>
mistery solved
< * jonasschnelli>
needs rest and waves goodby...
< jtimon>
and I'm sorry for being so biased in favor of zmq whenever it's an option for anything, I guess it may equilibrate some other biases against it.
< jcorgan>
one suggestion--we seem to be going back and forth between different use cases and different means to accomplish them
< jcorgan>
i'd rather clarify what all the use cases are first, then see what technology best suits them, then see how that maps to what exists and what doesn't in the client
< jcorgan>
*best suits each of them
< jtimon>
jcorgan: that sounds very reasonable
< jtimon>
analysis first, arch/design decisions later
< BlueMatt>
ugh, master is segfaulting trying to call rpc
< sipa>
BlueMatt: elaborate?
< BlueMatt>
gdb'ing
< sipa>
thanks
< BlueMatt>
but all i did was start and then try to call rpc :(
< BlueMatt>
oh, false alarm...custom change blew up
< BlueMatt>
sorry for the noise :/
< BlueMatt>
fucking 1-line change segfaults :(
< sipa>
we need a programming language with built-in reed-selomon code
< sipa>
so that the compiler can correct 1-line errors :p
< BlueMatt>
heh
< gmaxwell>
sipa: and updating the code means you need to read all of it?
< sipa>
gmaxwell: you mean you don't already do that?