< bitcoin-git>
[bitcoin] jameshilliard opened pull request #14564: Adjust configure so that only bip70 is disabled when protobuf is missing instead of the GUI (master...bip70-disable-check) https://github.com/bitcoin/bitcoin/pull/14564
< sipa>
achow101: see #14565... i tried implementing importmulti by recursing into all scripts and pattern matching, and it seems more concise than the special signer logic approach
< meshcollider>
sipa: I really like your importmulti cleanup
< bitcoin-git>
[bitcoin] kallewoof closed pull request #14507: net: avoid being disconnected from pruned nodes when syncing up (master...net-pruned-limit-requests) https://github.com/bitcoin/bitcoin/pull/14507
< sipa>
meshcollider: cool
< meshcollider>
sipa: is there a way to ToString() on a specific index of a ranged descriptor to get a concrete derivation path? Or does ToString() always only return the ranged one
< meshcollider>
looks like the latter
< meshcollider>
and if not, would it be sensible for me to add it? Or is there a better approach im missing
< Hayro>
hello
< sipa>
meshcollider: i thought about adding a way to 'specialize' a ranged descriptor to just a specific indix
< sipa>
but then i realized that that would actually be duplicating #14477
< bitcoin-git>
[bitcoin] ken2812221 opened pull request #14569: travis: Print characters per 9 min to avoid timeout (master...travis-avoid-timeout) https://github.com/bitcoin/bitcoin/pull/14569
< wumpus>
I'm getting really lost in all the wallet directory stuff to be honest
< wumpus>
didn't I look at another PR for exactly this shortly ago
< promag>
it was closed and replaced with this
< promag>
the other used path accessors and fs::equivalent which touches the filesystem
< promag>
this one only drops the base string from the path string
< promag>
and since we are recursively iterating walletdir there won't be errors, hence the assert() instead
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #14571: [tests] Test that nodes respond to getdata with notfound (master...Mf1810-qaNotfound) https://github.com/bitcoin/bitcoin/pull/14571
< wumpus>
well, sure, in the context you could say that it never happens, but if you define an utility function I think you need to handle errors properly and not assert
< wumpus>
or at the least add a comment and explain that the function will crash your program if the input isn't exactly as expected
< wumpus>
otherwise, before you know it, someone will use it in network code or whatever and you have a DoS
< wumpus>
documenting assumptions is very important
< wumpus>
and I think in general handling errors at the callsite (the decision can always be to crash) is better than crashing inside a helper function
< * wumpus>
really appraciates rust's Option<> and Result<> types it's so refreshing after seeing years of broken error handling hacks in C-ish languages
< promag>
wumpus: I can just inline the expression
< wumpus>
yes
< wumpus>
still, add a comment please
< wumpus>
why the assert isn't randomly going to crash the program for some user at some point
< karelb>
not sure if it belongs here - when I read this https://bitcoinops.org/en/newsletters/2018/10/23/ and the issues around remote RPCs, I realized this might be a problem if you run a browser on the same PC
< karelb>
since browsers now have localhost as a trusted origin, so you can connect to HTTP (without SSL) from any website
< karelb>
Evil website can try to guess bitcoind credentials and you have the same problems, that the article describes
< sdaftuar>
is there any reason to run a linter on an old branch?
< promag>
sdaftuar: keep consistency on backports?
< wumpus>
karelb: it's a drawback of using a tcp port for RPC I suppose, let alone http
< karelb>
wumpus: maybe httpserver.cpp should check an origin header and not allow cross-origin browser requests? (or whatever header browsers add)
< wumpus>
that might be a good precaution
< karelb>
I haven't actually tried that, maybe it won't work
< wumpus>
but only if you're sure this is actually a threat
< karelb>
(I mean I havenmaking a request from w
< karelb>
*I haven't tried making a request from a website
< karelb>
it's a similar threat as connecting from a different IP, no?
< karelb>
*IF* it actually work :)
< wumpus>
so are you sure browsers allow making json-rpc requests to localhost? this didn't use to be the case at least
< karelb>
wumpus: no, I am not sure, I just don't see why it would't work. Since it is just a http GET request.
< wumpus>
it requires a *post* request
< promag>
I think it's possible
< wumpus>
submitting a JSON-RPC command through get is impossibl
< karelb>
wumpus: what recently changed is that browsers allow connection to http from https websites, if the url is "localhost" (or "localost:port"). It is special-cased. Normally, you cannot make a request to http from https website. It's quite new (1 year, 2 years-ish)
< queip>
karelb: browsers just trust 127.0.0.1? so any JS on any site could talk to say https://127.0.0.1:7657 or such? that would be a critical vulnerability to many services that have localhost admin panels (with no password) no?
< promag>
this should be possible: fetch(url, { method: 'post', body: ... })
< queip>
karelb: is this really how it works? this means i2p, and many web-panel based servers are compromised now
< karelb>
well the service might restrict origin, or cross requests in general
< promag>
afak with the right cors headers it is possible
< karelb>
browsers behave well, they send the origin in some header
< queip>
I bet most do not... seems like idiotic move by browsers?
< karelb>
but it's solving the "inverse" problem - if a website can trust what it is fetching
< karelb>
anyway I will just try to hack something to test it, better than talk :D
< queip>
I have an issue with the github merge script, used in another (sort of private temp, but opensource overall) project... In one case (out of hundreds times using it with bo problem) now git diff HEAD~ in the merge shell shows nothing
< queip>
any idea what it might be? or, someone wants to look at the git with me? it might be some bug in the tool (or this time out of 1000+ somehow we're using it wrong)
< instagibbs>
wumpus, pyflake major version update in a minor flake8 update :shrug:
< wumpus>
instagibbs: :shrug: typical
< instagibbs>
we filed a complaint on their gitlab to at least have release notes
< queip>
wumpus: wouldn't you be interested by any chance to look at this git problem, which miiight be a problem in github merge script? I can't figure out why diff HEAD~ is empty... if no one is interested will just merge it, so might be not able to reproduce that later
< karelb>
OK I panicked too soon; browser does a CORS request and it fails, so it won't continue to connect further.
< karelb>
`Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://localhost:18444/. (Reason: CORS request did not succeed).`
< wumpus>
queip: if git diff HEAD~ shows nothing, you have an empty commit
< karelb>
and the website cannot even distinguish whether it's because the server is not running or whether it is not existent. So all it OK.
< wumpus>
karelb: good to know; thank you for trying
< queip>
wumpus: git log -p (int githubmerge.py shell) does indeed show various commits, that are not present on the PR-target branch normally, this is not a "doing nothing" MR. or did you ment that ONE of commits there might be empty and this triggers such reaction
< karelb>
(maybe it would be a good feature to allow that, to allow websites that interact with full node :D but that is beyond the scope)
< queip>
karelb: :]
< wumpus>
karelb: that's going to be very firmly rejected, I expect
< karelb>
you *could* whitelist, as a user, the websites that can connect to the full node. and the user would still need to give the website his name/password/cookie explicitly. so it would not compromise security. :P *but* I see what you mean :D
< wumpus>
yes, but no way
< karelb>
I see. :))
< karelb>
well, you can always write the web app in Electron, that ignore the CORS requests and can grab the cookie directly from the filesystem. Yum, Electron. OK, really out of scope now. :)
< wumpus>
this is getting too scary
< karelb>
javascript world is scary
< wumpus>
don't let it bleed into here
< Sentineo>
hn
< Sentineo>
but someoone might try to give you a browser which does not check CORS and connect to your bitcoin through rpc?
< Sentineo>
hopefully will not happen :)
< karelb>
once he already gave you his own binary on your pc, he can do worse...
< wumpus>
if someone 'gives you a browser' that is already the trojan horse scenario
< wumpus>
I think what discourages people from even trying browser-based attacks is that bitcoin-qt by default has the RPC server disabled, so most unknowning users won't be affected, the people that enable RPC will generally be more technical and hopefully be careful what they allow
< harding>
wumpus: someone in #bitcoin the other day said that RPC was enabled by default on Windows in bitcoin-qt. That surprised me, as I didn't even know bitcoin-qt could expose RPC (I thought you needed to use bitcoind for that). I didn't think to mention it here because I didn't realize that not running RPC with bitcoin-qt was supposed to be a security feature.
< luke-jr>
harding: RPC is intentionally enabled by default *on localhost*, and disabling it wouldn't provide any real security improvement I can think of
< achow101>
harding: it's supposed to only be enabled in qt if you have -server=1
< wumpus>
harding: setting server=1 in bitcoin.conf should be the only way to enable it with bitcoin-qt
< luke-jr>
I thought that changed with RPC cookies?
< wumpus>
no? not that I know, let's check
< gmaxwell>
I think we discussed that but didn't do it.
< achow101>
i don't think so
< gmaxwell>
wumpus: re 'gives you a brower' more like "this site only works in IE 6" (which has broken cross site requrest handing...)
< gmaxwell>
ethereum stuff has been raided over and over again with browser based attacks.
< harding>
Looked up the conversation on #bitcoin, the user didn't explicitly say that he was using the default config, so he could've had -server=1.
< harding>
I did test on Linux, and bitcoin-qt doesn't do RPC there by default.
< gmaxwell>
I don't think we really should be counting on the default config to protect people... lots of people copy and paste configs.
< gmaxwell>
echeveria crawled the internet and found something like 3000 bitcoin rpc ports listening, which would be a substantial percentage of all p2p listening nodes.
< wumpus>
I'm not *counting* on anything, I was just mentioning that it's not a problem with the default confi
< harding>
gmaxwell: I crawled too and only found 1,100.
< wumpus>
it shouldn't be a problem with RPC listening on localhost either, it's just defense in depth I suppose..
< wumpus>
if you don't need a RPC server it's better to disable it
< wumpus>
even if there is no way you can imagine it can be exploited
< wumpus>
in any case I checked: bitcoin-qt still doesn't enable RPC server by default
< harding>
wumpus: thanks. Sorry for the false alarm.
< gmaxwell>
harding: so 10% instead of 30%. :P
< harding>
gmaxwell: yeah, it was 13% of my sample.
< phantomcircuit>
harding, that is significant
< harding>
phantomcircuit: I agree.
< harding>
phantomcircuit: I wish I knew the cause. The only instructions I've seen for enabling it were from a particular mobile wallet, but I have a hard time believing there are over 1,000 mobile wallet users who also run a full node and also opened RPC to use them together.
< queip>
wumpus: the problem is in PR https://github.com/userghmrt/testrepo/pull/3 . Although on unpatched githubmerge.py it's needed to replace/steart def tree_sha512sum() function with return "0" (because repo uses symlinks) . I confirmed, there in github on that issue 3 git dif HEAD~ is empty evne though on GH page "Files changed: 1"
< queip>
btw you have invite to edit that test repo if you need to test
< gmaxwell>
harding: I think there are a lot of example config files floating around.
< phantomcircuit>
harding, the cause is pretty clearly people being told they need a specific config
< gmaxwell>
I know from past expirence that joe-user when faced with discovering they need a config file, go and paste some example. When I've asked users for their configs, they have in the past frequently given me ones copy and pasted from the wiki.
< harding>
gmaxwell, phantomcircuit: that seems likely.
< queip>
btw, we've patched github merge to support symlinks, submodules (incluees them in tree hash) and gitlab, if anyone wants at some point
< harding>
I don't know what the solution for that is, though. Better documentation and sample configs provided from an official source?
< gmaxwell>
harding: probably making bitcoin make its own template config file when one doesn't exist would help.
< harding>
I guess when .bitcoin is created, a default bitcoin.conf could be created with some normal-to-change options and basic comment documentation. I think you'd want to keep it short, rather than providing every possible option, so that people aren't tempted to delete the whole thing and replace it with a random config from the Wiki.
< gmaxwell>
I had assumed that problem had gotten less bad because of cookies -- you don't need to make a config file to make bitcoind usable at all... it may be that your 1k rpc listeners have a lot of nodes that came up before cookies existed.
< harding>
gmaxwell: interesting thought, and something that seems not to hard to check---for an open port 8332, get the node version for port 8333.
< gmaxwell>
I mean they could have installed 0.11 or whatever and since upgraded but already have a config.
< harding>
gmaxwell: oh.
< gmaxwell>
So this would be another advantage in adding an additonal config option that needs to be set to listen to the internet.
< harding>
Yeah.
< luke-jr>
my PR kindof does that
< luke-jr>
(they have to specify rpcbind explicitly)
< wumpus>
solution: delete all RPC binding functionality, switch to UNIX sockets
< luke-jr>
that doesn't work on Windows
< wumpus>
like c-lightning
< wumpus>
windows has local sockets as well (called differently) IIRC
< gmaxwell>
wumpus: would be nice, but perhaps too hard to get random software stacks to speak domain sockets.
< wumpus>
heck, windows 10 even has actual UNIX sockets
< gmaxwell>
luke-jr: Did you consider doing something more explicit? e.g. making an option called enable-insecurely-exposing-rpc-to-network=1?
< wumpus>
gmaxwell: I know right! should have done that from the start like c-lightning :-(
< wumpus>
so many things are imposslbe to change now
< wumpus>
queip: ok I'll have a look
< luke-jr>
gmaxwell: no. it's not necessarily insecure, if it's a private LAN
< luke-jr>
wumpus: can normal Windows software bind to UNIX sockets?
< gmaxwell>
luke-jr: enable-potentially-inscure-rpc-network-exposure? the point being so you can't just copy and paste without getting a warning.
< wumpus>
luke-jr: yes I think so
< gmaxwell>
wumpus: even there, my concern isn't so much intertia-- if it was inertia I think we could just do it and include some shim utility... but like, can nodejs applications ever speak to domain sockets?
< luke-jr>
gmaxwell: might be worth considering. I expect we'll find other problematic copy-and-paste configs though..
< wumpus>
we don't even have UNIX support for RPC yet, let alone could set it as only option :(
< gmaxwell>
I think it's okay if for good reason we introduce some incompatiblity in a major version, esp if we give people a long time to switch first... but it wouldn't be good if there is no easy way to become compatible with it. :)
< gmaxwell>
wumpus: indeed. well we certantly could do that.
< wumpus>
I'm sure nodejs can use any system API, it's a full environment for server software
< gmaxwell>
luke-jr: the worst is rpcpassword, but hopefully cookies reduced that.
< wumpus>
it might be in javascript but it's not *that* much of a joke
< queip>
would it be possible to tell users what is the problem, other than generic 403 message? like, tell them the option to add but warn them of consequences. or users will complain that "it stopped working after update"
< wumpus>
you can send any message with the 403 status, even have a full error html document
< queip>
actually it will not even listen on it to reply 403, just no connection by luke's default?
< gmaxwell>
Brendan Eich would say never underestimate javascript... which I always took to mean never underestimate the ability of something javascript to be a joke.
< wumpus>
though I don't think *the latter* is a good idea because software will try to parse text/html replies as json
< wumpus>
gmaxwell: +1
< wumpus>
a lot of software uses local pipes for RPC mechanism
< wumpus>
including databases, which nodejs people love
< gmaxwell>
queip: sure, our normal practice for depricating RPCs is to make them return errors and have an option to reenable them for one version. So the normal sequence we use where possible is (1) support the new thing (2) announce the old thing is going away [time passes] (3) disable by default the old thing but with a switch to override [time passes] (4) take out the old thing.
< gmaxwell>
So I think we could change this assuming things actually could talk to it. I don't know that we need to (like adding increasingly agressive warnings against enabling rpc, shifting more stuff onto domain sockets, etc may be enough...
< wumpus>
FWIW ssh, and I think stunnel, allow forwarding a UNIX domain socket, so it doesn't even make it impossible to expose the RPC over the network, just requires actual setup (and thinking about security)
< gmaxwell>
if stunnel can do it then that even gives us a backwards compt method if we ever drop TCP support.
< wumpus>
queip: yes, can you temporarily give me acces to the repo? I can work around it, as I don't actually need to push anything to test, but it's easier
< wumpus>
just checked: yes, stunnel supports a UNIX socket as destination, but only on UNIX; nginx also supports forwarding requests to UNIX socket based http servers
< wumpus>
queip: oh nm I have an invite
< jarthur>
wumpus: heard any word from libevent crew on your pre-existing-fd PR?
< wumpus>
jarthur: they wanted a different solution at the time; which was above my head, certainly, don't know if that ever made any progress
< wumpus>
the only functionality I needed was to inject an existing fd into the http client, same as they allow for the server binding
< esotericnonsense>
hm
< esotericnonsense>
i've just jumped in here and stuck some chat in #bitcoin
< esotericnonsense>
yes, CORS should (on behaving browsers) prevent requests to localhost and also prevent requests on different ports
< esotericnonsense>
so _even if_ you have some service running on localhost:10080 say, and you're at http://localhost:10080, you shouldn't hit it (that is if CORS is still enabled for http, i don't really use bare http sites, but I'd assume why not)
< esotericnonsense>
a bad browser can just ignore CORS but then that's just equivalent to having an insecure system, you're running vulnerable code
< esotericnonsense>
personally i'd probably ask why bitcoin-qt even has the ability to enable RPC
< esotericnonsense>
it feels like a 'nice to have' footgun
< esotericnonsense>
is anyone actually seriously running the GUI client as their backend
< wumpus>
what, if you want to enable RPC in bitcoin-qt you should be able to
< esotericnonsense>
what's the use case?
< wumpus>
I don't believe in making things hard to do because there are a few people that do stupid stuff with it
< gmaxwell>
esotericnonsense: it enables you to do things like use joinmarket.
< luke-jr>
esotericnonsense: doing stuff from the commandline.. or third party apps
< luke-jr>
eg, I think many people probably use it for their taxes
< wumpus>
punishing power users for the mistakes of others
< gmaxwell>
If there were litterally no usecase we could come up with then I would probably agree with esotericnonsense's point, but there are many. :P
< wumpus>
I have various scripts that interface to bitcoin and I use them with the GUI too.
< esotericnonsense>
i suppose this is just part of my general 'why is the gui still linked in' grumble but then I stopped working on it so can't complain :P
< wumpus>
sigh...
< wumpus>
I'll just shut up, I end up disagreeing with everyone on everything anyway
< luke-jr>
if GUI had to access over RPC, then RPC would become mandatory for GUI users ;)
< esotericnonsense>
(not that it would help especially I guess since someone copying a random config could just as easily copy a random config in to their bitcoind instance)
< wumpus>
if it's any guide, even professional stock trading software has a way to enable a RPC interface in their GUI; there's simply users that want to control a program programmatically even if it's a GUI program
< sipa>
wumpus: please don't shut up :)
< wumpus>
in some operating systems there is hardly anything *but* GUI programs
< esotericnonsense>
luke-jr: sure but it wouldn't have to be tcp. i don't know enough about cross platform sockets to comment properly though. the default could be that the bitcoind and qt processes are started with a user that sets permissions on the socket
< esotericnonsense>
but that's just unix, i've no idea how this works on win
< jarthur>
I run the GUI client as my backend all the time, though am probably a "power user". Same with Trader Workstation, as wumpus alludes.
< wumpus>
you might have completely the wrong idea of what people use a GUI, it's not only clueless people
< esotericnonsense>
wumpus: I don't think it's the case that only clueless people use a GUI
< esotericnonsense>
at all, sorry if it seemed that way
< * harding>
loves the GUI, but also loves CLIs for almost everything else
< esotericnonsense>
more that it seemed using the GUI as the GUI, and the backend as the backend makes sense, but of course you currently can't do that, so I'm just ranting, basically :P
< esotericnonsense>
if this is actually a problem, could there be a warning sign or something in the corner of the gui that says 'rpc is enabled, did you know?'? (maybe it already exists)
< wumpus>
esotericnonsense: I think that's a good suggestion, no that doesn't exist, feel free to make an issue!
< wumpus>
could certainly have an icon for that
< luke-jr>
IMO sensible defaults + education is the solution for things like this
< esotericnonsense>
i've just sort of popped in and out on this but i've seen the masses of open RPC ports mentioned a bit, do we have any idea why this is the case?
< * esotericnonsense>
should make a PR for an alert RPC
< harding>
esotericnonsense: the guess above was that people are copy/pasting configs that do things they don't necessarily need.
< esotericnonsense>
connect to all of them and say 'oi, you should probably not do this, especially with password:password' :P
< wumpus>
'alert' RPC hehe
< wumpus>
but I think a good point is that having RPC listening is currently effectively hidden to the user, it's also not configurable from the GUI afaik, only by editing the conf
< luke-jr>
maybe RPC should accept an 'alert' RPC without password :P
< esotericnonsense>
luke-jr: that was my thought
< wumpus>
uhm no
< esotericnonsense>
i mean if you wanted to get really clever with this
< esotericnonsense>
you could build in to the p2p network that clients attempt to connect to each others' RPC and issue the 'kill RPC' command if it's publicly routable
< esotericnonsense>
lol
< esotericnonsense>
now i'm just having too much fun ;P
< Sentineo>
:)
< Sentineo>
they would have to signal the killme bit :)
< esotericnonsense>
well if the alert command is behind rpc auth
< luke-jr>
if it's sufficiently secure, it's useless for notifying people with the port exposed
< esotericnonsense>
yeah
< esotericnonsense>
well, not quite.
< esotericnonsense>
if bad passwords are brute forced then it might be useful.
< wumpus>
of course it's dangerous, then again, so are many other RPC commmands, this would be useful to communicate from say, a backend script to the UI… but I don't htink it's actually a good idea just wanted to show how easy it is to do
< gmaxwell>
hm. so one thing that could be done in the GUI is the first time the rpc is connected after you start, don't allow the connection until the user confirms a dialog.
< esotericnonsense>
(i'd consider RPC access as probably RCE anyway though I suppose it would require a determined attacker)
< gmaxwell>
esotericnonsense: part of the reason we don't want the RPC internet exposed is because we don't want YET ANOTHER vector for unauthenticated RCEs.
< wumpus>
you could do the same for REST, with a pre-programmed message, if you want it to be available with less security
< esotericnonsense>
gmaxwell: yes, exactly, I think i'm being misinterpreted, sorry
< * midnightmagic>
secretly merges alert rpc
< wumpus>
midnightmagic: it reminds me of the fun when windows had this built in
< esotericnonsense>
what I mean is that authenticated alert being used to send "install this badness.exe" seems irrelevant if authenticated rpc means you own the box anyway
< esotericnonsense>
unauthenticated alert is bad sure
< wumpus>
midnightmagic: you could make *any* computer pop up an alert xD
< esotericnonsense>
that said, unauth alert could just give a predefined message
< midnightmagic>
wumpus: death on flaxen wings :-)
< gmaxwell>
E.g. "A remote control connection is being attempted to your wallet. If you did not initiate this action rejected it. [Allow remote control during this session.] [Reject this attempt.] [Disable remote access].
< gmaxwell>
"
< aj>
gmaxwell: needs a "[ ] Always accept these requests" checkbox...
< wumpus>
yes, so, this is even more dangerous for the wallet
< wumpus>
which is another reason why having the wall... ok never mind
< gmaxwell>
aj: why?
< wumpus>
I kind of like how monero doesn't have a remote API, or daemon for the wallet at all, it's just a command line tool that connects to the node
< aj>
gmaxwell: sorry, being sarcastic because i hate permission dialogs
< gmaxwell>
all the examples I gave above about rpc being needed were wallet examples.
< esotericnonsense>
aj: it would be Always accept these requests by default
< esotericnonsense>
> gmaxwell | hm. so one thing that could be done in the GUI is the first time the rpc is connected after you start
< wumpus>
I don't think popping up a dialog for for the first RPC request is a bad idea btw
< esotericnonsense>
i suppose once per application start is distinct from once per... ever
< wumpus>
that's, kind of, how those things usually work
< gmaxwell>
It's one of the things you can do with the GUI... you can get user interaction.
< wumpus>
yes
< esotericnonsense>
once per IP address (ever) seems reasonable
< gmaxwell>
esotericnonsense: meh, ever requires remebering it, and also ends up being less secure.
< wumpus>
nah first make the RPC localhost-only
< gmaxwell>
if it's a nussance, there could be a config override.
< wumpus>
that's like low-hanging fruit
< wumpus>
after that, you could add interaction
< gmaxwell>
the non-interaction improvements are needed to protect bitcoind in any cas.e
< wumpus>
but *remote* RPC is just a stupid idea
< gmaxwell>
I wouldn't be surprised if most of those rpc listners are bitcoind... after all, bitcoind needs some use of the rpc regardless.
< wumpus>
yess I know how stupid that sounds as RPC stands for Remote Procedure Call
< wumpus>
right
< esotericnonsense>
yeah i mean personally i don't see why it should be able to listen outside of localhost (or just, socket preferably)
< esotericnonsense>
even though it would probably break All The Things
< wumpus>
it's kind of a no-brainer, I'm just afraid of hordes of users complaining about Breaking Things
< jarthur>
Any time I've done remote RPC (whether with SSH tunnel, or plain old terrible direct connection), it was because one server had enough hard drive space and the other did not.
< jarthur>
or I was testing something
< wumpus>
I've actually used remote connections for valid reasons (but yes, always over SSH tunnels)
< esotericnonsense>
about the only valid config I can think of right now apart from _really_ obtuse cases like a raspberry pi connected directly to another box without a switch
< esotericnonsense>
would be rpcallowips with wireguard
< esotericnonsense>
so that it's actually authenticated remote rpc
< wumpus>
there's also the multi-VM scenario where network connections are inherently secure because the network is virtual
< gmaxwell>
the problem with disabling non-local is things like private networks between hosts, which are perfectly fine assuming its setup correctly.
< wumpus>
but yeah...
< gmaxwell>
and indeed VMs.
< gmaxwell>
We could set the TTL on those connections to 1. :P
< esotericnonsense>
i mean localhost in general bugs me.
< echeveria>
just disallow binding to 0/0.
< esotericnonsense>
that it's a tcp socket is already 'everyone on this box'. the remote within "trusted" LAN situation is essentially the same
< wumpus>
with docker it's incredibly common to have one application per container and have them communicate over point-to-point virtual networking, I'm sort of afraid of making those use-cases impossibl
< echeveria>
require binds to be explicit.
< gmaxwell>
esotericnonsense: yes, localhost is also a problem, but short of domain sockets we can't really do better than localhost + auth.
< esotericnonsense>
wumpus: the docker case is reasonable yeah, as with VMs. a proper ethernet bridge.
< echeveria>
this prevents copy-paste configs from working, massively reduces the ability for bad configurations, but still allows for usability over VPN tunnels, etc.
< gmaxwell>
echeveria: interesting point, normally you don't want to beceause addresses change, but all the "okay" examples I gave above, they don't.
< esotericnonsense>
the case in which you have boxA and boxB on the bridge and _noone else_, nice,.
< wumpus>
yes luke-jr's PR makes sense, to at at least make binding explicit
< esotericnonsense>
binding to an IP explicitly breaks docker. I think. unless you script it.
< gmaxwell>
luke's patch also changes it to how I already thought it worked.
< esotericnonsense>
no idea what IP a container will get.
< jarthur>
link to luke's PR?
< harding>
#14532
< gribble>
https://github.com/bitcoin/bitcoin/issues/14532 | Never bind INADDR_ANY by default, and warn when doing so explicitly by luke-jr · Pull Request #14532 · bitcoin/bitcoin · GitHub
< jarthur>
ty
< esotericnonsense>
i suppose localhost + cookie, if the cookie has the correct permissions set, is similar to a permissioned domain socket
< esotericnonsense>
the issue is really the strength of the auth mechanism and basically whether the RPC is safe pre-auth
< esotericnonsense>
'open port' doesn't really matter if that _actually works_
< esotericnonsense>
(as in, the situation post 14532 seems reasonable)
< echeveria>
esotericnonsense: the RPC server is unlikely unsafe pre-auth, it's at least a denial of service risk.
< jarthur>
gmaxwell: huh, it's funny, that's how I thought it worked already too
< jarthur>
esotericnonsense: one thing to consider with permissioned domain sockets is Linux abstract unix domain sockets have similar "openness" as TCP loopback.
< esotericnonsense>
jarthur: eh? it's not possible to listen on a specific user, right?
< jarthur>
With loopback, you at least have some decent control via netfilter/iptables/ufw on Linux
< esotericnonsense>
(for TCP loopback)
< esotericnonsense>
the domain socket can be rw only for the user
< esotericnonsense>
e.g. if you had bitcoind, bitcoin-qt and bitcoin-qt had perms on the socket but other than bitcoin-qt only root did
< wumpus>
!action merge #14532 I guess
< * gribble>
merge #14532 I guess
< jarthur>
esotericnonsense: it can't if it's a Linux abstract unix domain socket
< esotericnonsense>
oh sorry. missed 'abstract'.
< booyah>
are people actually losing money / being hacked, as result of many listening on internet?
< luke-jr>
Typically exploits are fixed even if nobody has been compromised with them yet
< booyah>
sure, not suggesting it's not something to be fixed, just wondering what is the situation
< gmaxwell>
not that we're aware of, but I suspect most of us are uncomfortable with all this rpc exposure.
< gmaxwell>
the additional attack surface is a liability we don't want to deal with.
< wumpus>
how would you even know if people are losing money through this
< jarthur>
It was real embarrassing for us on the Electrum project when the localhost auto-bind we were going to get around to at some point wasn't gotten around to until it hit the media. :)
< booyah>
if someone would reported that happening, wumpus. ofc most would not
< esotericnonsense>
well the fact that it's now known increases the probability of exploit too, right.
< wumpus>
also agree with luke-jr, it's better to prevent in this case
< gmaxwell>
booyah: yea, problem is that people don't know or don't report. I'm somewhat doubtful it's causing much loss right now but it means that any rpc/rest bug is potentially much more serious.
< luke-jr>
jarthur: what port does Electrum bind?
< gmaxwell>
at least given what we currently believe: that it's being enabled due to accident/confusion.
< luke-jr>
I wonder if these 8332s aren't even bitcoin itself?
< esotericnonsense>
luke-jr: is it possible to get a bitcoin-like response back with invalid auth
< esotericnonsense>
could just hit a few (or a lot) of them and check
< * esotericnonsense>
checks his node
< luke-jr>
has anyone done so?
< booyah>
do we have the IP list? anyone proceessed it, are theses opened e.g. mostly in some specific network, or OS? (maybe there's a product or something that does that)
< wumpus>
maybe it's a honeypot
< harding>
luke-jr: the list of IPs I scan cam from bitnodes, which I think currently filters out Bitcoin Cash nodes. Obviously so were probably spy nodes and the like.
< harding>
s/so/some/
< luke-jr>
harding: well, I mean perhaps some other software is listening in 8332 while Bitcoin isn't
< wumpus>
or maybe they found an exploit in bitcoin-cli and are waiting for you to use it on them :-)
< esotericnonsense>
lol. I forgot, it's obvious
< esotericnonsense>
curl localhost:8332
< esotericnonsense>
JSONRPC server handles only POST requests
< gmaxwell>
wumpus: lol
< esotericnonsense>
so yeah if you have the list it would probably take a few seconds to hit them all and see if they're not bitcoin. :P
< jarthur>
luke-jr: electrum's RPC port is randomly selected at startup if you don't configure one. Still didn't take long for scans to pick them up though.
< wumpus>
yeh randomizing ports helps against attackers looking for targets, not against anyone that already selected you
< esotericnonsense>
luke-jr: what might be interesting
< esotericnonsense>
is scanning all IPv4 for 8332 (i.e. hosts that don't listen on 8333)
< wumpus>
e.g. you can't currently scan the whole IPv4 range for all ports
< wumpus>
right
< esotericnonsense>
you can disable listening whilst leaving rpc up right? :P
< wumpus>
oh sure
< esotericnonsense>
the odd person might have even fat fingered 8332 instead of 8333 in their dnat :P
< luke-jr>
that sounds more likely
< gmaxwell>
you can scan all ports on all hosts that visit your website/irc channel/etc. however.
< wumpus>
yes, I guess the threat model for bitcoin is somewhat different than say, ssh
< booyah>
but there are people where actually something respons to TCP 8332 right? so not just opened firewalls / forwarded ports
< esotericnonsense>
wumpus: heh you've reminded me that I have a weird memory-like error in ssh at the moment... my linux-fu is insufficient to debug it
< esotericnonsense>
ssh host cat /dev/zero > /dev/null gives me an EFAULT in strace, read into a bad memory address
< esotericnonsense>
anyway, this is not bitcoin core dev, sorry :P
< wumpus>
I'd think it's more likely for the crash to be the result of some over-protective security feature in ssh than a security bug, but yeah
< luke-jr>
I don't see how
< luke-jr>
cat /dev/zero shouldn't have any security implications
< esotericnonsense>
the system call that fails is a read from the TCP socket (checked fd)
< gmaxwell>
Is there a socket option to set the TTL on tcp connections? If there is we should perhaps make the rpc use TTL=1 unless overridden by a config option.
< esotericnonsense>
gmaxwell: ooh!!
< esotericnonsense>
that is really, really nice.
< esotericnonsense>
if it's possible.
< luke-jr>
good idea
< echeveria>
esotericnonsense: it's fairly trivial to scan 0/0 for a specific port now. there's port scanners that run directly on the NIC which can saturate multi gigabit links.
< gmaxwell>
that nicely covers the 'lan/vpn good / internet bad'.
< echeveria>
that said, I'd expect all ports listening on 8332 are listening on 8333.
< esotericnonsense>
echeveria: yeah I'm aware of that, hence me thinking it might be interesting to see what's on 8332 "globally".
< wumpus>
if there is one I've never heard of it, you can usually set it on a OS level but not per socket IIR
< esotericnonsense>
echeveria: probably yeah.
< wumpus>
it's a fascinating idea though
< aj>
ip(7) IP_TTL sounds plausible?
< esotericnonsense>
man 7 ip; IP_TTL (since Linux 1.0)
< esotericnonsense>
yes
< esotericnonsense>
you want it to be multiplatform though, :P
< jonasschnelli>
The current DMG provided in bitcoincore.org/bin does not open on macOS
< wumpus>
agree, would make sense to make a 0.17.0.1 for macosx only
< luke-jr>
jonasschnelli: the only other fix currently in the branch is so minor, it wouldn't make sense to make a new branch for 0.17.0.1
< jonasschnelli>
We can also directly move to 0.17.1,...
< sipa>
that seems in line with versioning we've used before, using the 4th number for platform specific fixes
< luke-jr>
doesn't MarcoFalke have a bunch of fixes backported to 0.17 though? might make more sense to just move on 0.17.1
< luke-jr>
sipa: sure, I'm just saying we can tag it on 0.17 branch
< sipa>
some of those may be nontrivial; i saw some issue with his backports PR having a merge conflict?
< cfields>
luke-jr: that would mean that Mac users couldn't drop back to 0.17 if 0.17.1 was buggy.
< cfields>
+1 for 0.17.0.1
< luke-jr>
cfields: 0.17.1 should only be fixes on top of 0.17.0 anyway
< wumpus>
sipa: yep
< promag>
we could pick the simple backport fixes (including macos fix) and let the remaining for 0.17.2?
< wumpus>
I dont' think we have enough for 0.17.1 yet
< wumpus>
if a lot of people are experiencing issues with 0.17.0 on MacOSX we should do 0.17.0.1 soon
< wumpus>
like, tomorrow
< sipa>
agree
< jonasschnelli>
ack
< cfields>
sgtm
< jonasschnelli>
I think it's not an urgent thing,.. but it may fraighten off users since it can cause a finder crash
< * luke-jr>
shrugs
< cfields>
jonasschnelli: now this is interesting!
< jonasschnelli>
Someone told me Apple is aware...
< cfields>
heh, ok
< jonasschnelli>
A finder crash smells just really bad and AFAIK there has been some exploitable bugs in that area in the past.
< wumpus>
it was funny hwo this is caused by a python unicode versus bytes issue
< jonasschnelli>
Indeed!
< jonasschnelli>
I just don't get why it was a non-issue when compiling with trusty.. I though we had the same python version.
< jonasschnelli>
*thought
< jonasschnelli>
however,.. lets just do a 0.17.0.1 macOS asap
< wumpus>
it's great that you managed to isolate it
< jonasschnelli>
Took me around 30 gitian builds. :)
< cfields>
yes, thanks for that. I always check that dmgs open before tagging the detached sigs, but I've stayed on 10.11 to catch back-compat issues, so I guess I avoided it :\
< jonasschnelli>
Yes. I also take it on me,... I haven't done that. But non of us somehow did test the DMG during all the RCs...
< jonasschnelli>
which is a bit strange
< sipa>
that's a bad sign
< wumpus>
not many people testing on macosx?
< cfields>
indeed. Maybe we could start adding "Tested ACKs" for the gitian sigs PRs
< cfields>
to at least verify that someone has started up the release
< wumpus>
I can only test the linux release myself
< wumpus>
(and I test compiles on FreeBSD and OpenBSD! but that's irrelevant to gitian)
< jonasschnelli>
At least fanquake, sjors and other gitian builders use macOS regularly... including myself. But meh,.. I don't know why I haven't detected it
< luke-jr>
testing RCs is IMO the job of users, not builders or coders
< booyah>
I can get you mac os X (old macbook) or bsd test (install+run) if you want
< jonasschnelli>
Both I'd say
< cfields>
I'll at least start ACKing the platforms that I've verified startup for the sake of posterity.
< sipa>
luke-jr: agree, but no reason why someone can't be both - and regardless of what you call them, not enough people testing rcs is concerning
< luke-jr>
sipa: right, my point is that PRs isn't a good place for it
< luke-jr>
users don't make PRs
< sipa>
that's reasonable... though how else do you get feedback?
< luke-jr>
short of writing a website that collects it, and asking with the RC announcement to post there.. coming up blank
< wumpus>
usually we ask people to open issues on github for problems
< luke-jr>
maybe bitcointalk can offer a forum specifically for testing reports or something
< wumpus>
I'm nto sure a different website is needed
< luke-jr>
true
< luke-jr>
what's missing is *positive* feedback
< wumpus>
reddit etc have too much noise
< wumpus>
yes, true
< luke-jr>
and apparently the users testing to give it
< luke-jr>
maybe the -core-dev ML needs to get more attention from users so they learn about and try RCs?
< wumpus>
I.. dont' think you can really get users into a ML these days
< sipa>
haha
< luke-jr>
how do users learn about stuff now?
< sipa>
maybe we need a facebook group *ducks*
< luke-jr>
>_<
< wumpus>
:-)
< cfields>
luke-jr: real answer: software force-updates itself.
< cfields>
:(
< wumpus>
twitter is really popular yes
< luke-jr>
how about I make a Twitter account for posting experimental releases of Core, Knots, and other reputable Bitcoin projects? (Electrum, etc?)
< luke-jr>
or is there some kind of shared Twitter account thing so more than just I can post?
< luke-jr>
cfields: not to testing versions..
< sipa>
we can tweet from the bitcoincoreorg account
< luke-jr>
sipa: not everyone wants to see experimental releases
< aj>
luke-jr: could have a flag to say "auto update to current release version" and another to say "follow experimental release candidate stream"
< achow101>
no auto update pls
< warren>
luke-jr: have a different twitter account for those interested in following RC's
< luke-jr>
warren: that's what I said :P
< wumpus>
right, at least at the moemnt we don't tweet experimental releases from bitcoincore twitter
< sipa>
yeah
< wumpus>
I don't even post them to my personal account ,maybe I should
< luke-jr>
wumpus: that might be enough
< warren>
Thought you migrated permanently from Twitter. =)
< luke-jr>
XD
< sipa>
remind me, i can tweet from my personal account too
< wumpus>
warren: well I'm more comfortable posting development-related stuff on mastodon, but for noticifcations to reach as many people as possible I'd certainly use twitter
< warren>
(What is the protocol for topic proposals? wait until this topic is done?)
< wumpus>
warren: no, just propose
< wumpus>
ideally you propose topic subjects at the beginning of the meeting
< wumpus>
then we can cycle through them
< wumpus>
but it's fine we're done with this now
< sipa>
i wanted to bring up the linter issues
< warren>
topic proposal: Interested in opinions regarding the risk of bringing back Fortuna. Along with deprecation of BIP70, we are on the path toward eventual removal of the openssl dependency.
< sipa>
we really don't need fortuna or a high-performance built-in randomness pool - we don't need randomness frequently
< sipa>
what we do need is a good way to seed entropy from the environment
< rex4539>
I think it is quite unlikely that "normal" people will ever run an RC because they are afraid of losing funds because of bugs. Others like me, on the other hand, only run beta software. If something is "stable" I don't want it. I want experimental, unstable software full of bugs. More fun. I believe the reason for missing the DMG bug is that everyone here is building from master and doesn't actually run the public builds.
< rex4539>
Perhaps there should be a pre-release QA checklist for basic functionality on all supported platforms.
< wumpus>
#topic Fortuna
< wumpus>
—or other randomness
< warren>
well, Fortuna or the lesser goal of seeding entropy from the environment
< sipa>
and seeding entropy from the environment is annoying as it's a platform specific business
< sipa>
but we have a built-in randomness pool now - it's not fast, but it's more than good enough for what we need
< sipa>
it's just seeding through OpenSSL mostly
< warren>
I am encouraged that #14451 happened, deprecating BIP70 (huge attack surface, nobody uses it etc.) This means we will eventually be able to remove the openssl dependency. Except for that part.
< gribble>
https://github.com/bitcoin/bitcoin/issues/14451 | Add BIP70 deprecation warning and allow building GUI without BIP70 support by jameshilliard · Pull Request #14451 · bitcoin/bitcoin · GitHub
< sipa>
and whenever we need "strong randomness" we mix data from openssl and our own pool
< sipa>
i think it's sufficient to have a c++ file with a bunch of entropy gathering things in it, without turning it into a C API or whatever
< sipa>
warren: nah, i think that's total overkill now
< jonasschnelli>
Just read /dev/urandom? *duck*
< warren>
external API is risky as you need to worry about about fork safety and conditions you can't predict
< gmaxwell>
jonasschnelli: and then not get randomness when FD exhaustion means you can't open it.
< sipa>
jonasschnelli: we already do that
< sipa>
warren: not if it just gathers some entropy from the environment (and not a full RNG library)
< warren>
Do we already use the newer syscall that blocks if the kernel prng is not seeded?
< sipa>
yes
< kanzure>
how do the proofs of randomness/entropy incorporation work
< jonasschnelli>
gmaxwell: don't we just need a single seed during startup then ChaCha20 PRNG from. there? But i'm not familiar with the details..
< wumpus>
yes we use the syscall where available
< warren>
I know a lot less about the other Unixes.
< wumpus>
on Linux and various BSDs
< sipa>
jonasschnelli: no, that's FastRandomContext
< sipa>
and it's only used to generate randomness that doesn't need independently seeding
< jonasschnelli>
I see
< gmaxwell>
jonasschnelli: only if everything works right, users never use virtual machines with snapshooting, and we don't care about being totally broken in the face of OS bugs like netbsd and freebsd had in the last couple years.
< sipa>
for generating private keys etc, we gather new entropy every time
< warren>
----> <sipa> i think it's sufficient to have a c++ file with a bunch of entropy gathering things in it, without turning it into a C API or whatever <--- This would be good enough and people would feel it is worth the risk of change to be able to eventually remove the openssl dependency?
< gmaxwell>
(and don't mind a later process memory leak potentially revealing the keys we previously generated, etc)
< gmaxwell>
Until BIP70 is gone we're stuck with openssl regardless. we lost urgency on discontinuing using openssl as a randomness input after bitpay started requiring BIP70 to make payments.
< gmaxwell>
So long as we have openssl for other things it's a harmless addition to random inputs.
< cfields>
gmaxwell: IIRC that's only supported in -qt
< sipa>
ah, for "non-strong" randomness (GetRandBytes, as opposed to GetStrongRandBytes) we use just OpenSSL, would people be ok with switching that to our own randomness pool?
< gmaxwell>
we do? well that should be switched.
< jonasschnelli>
Agree
< sipa>
GetStrongRand uses all sources available, and mixes them into our own pool
< sipa>
including OpenSSL
< wumpus>
yes
< wumpus>
switching over non-strong randomness is a no-brainer
< jarthur>
topic proposal: Unix socket support for RPC API
< wumpus>
eh <3 minutes left
< jarthur>
No prob. Next time.
< wumpus>
I agree though
< wumpus>
FWIW
< wumpus>
need that yesterday!
< aj>
missed out on the linter stuff?
< sipa>
guess we'll do that another time
< sipa>
or hope it resolves itself
< wumpus>
linters are great fun for the whole family, you can watch them fail in real time in so many different ways
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Oct 25 20:00:27 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< cfields>
Quick takeback: I wasn't entirely ready yet, and jumped the gun on coming back to dev after some time off a few weeks ago. Sorry if I've slowed anything down since then as a result, and for the annoying false start. Please don't let anything block on waiting for me in the near future.
< cfields>
That's probably implied by now, just wanted to be explicit about it.
< wumpus>
cfields: thanks for anything you've done, and no pressure for anything else
< cfields>
wumpus: I'll for sure take care of whatever's needed from me for 0.17.0.1
< cfields>
though I suppose that's nothing, if it's mac only :)
< wumpus>
usually we build the release for all platforms I think, even though there's no effective chagne for others
< wumpus>
but e.g. the website doesn't support hosting one version for one platform and another for another
< cfields>
Makes sense, avoids having to customize all of the build scripts and stuff
< cfields>
ok, will handle that as usual then.
< wumpus>
do we really need release notes for 0.17.0.1?
< cfields>
no opinion
< wumpus>
nah just have to mention the macosx issue I guess
< sipa>
yeah
< jnewbery>
cfields: take care of yourself first. Everyone here is very grateful for the enormous contributions you've made.
< jnewbery>
sorry I missed the meeting. On the topic of testing RCs, the optech newsletter has action items each week, which included 'allocate time to test Bitcoin Core RC' for the weeks that we had RCs (eg https://bitcoinops.org/en/newsletters/2018/09/11/). If there are more specific instructions, let us know and we can help spread them.
< wumpus>
jnewbery: +1
< gmaxwell>
kallewoof: Are you or anyone else working on getting the "spend all inputs to an address at once if it isn't more expensive" thing going again?
< sipa>
I dont't Optech members to be the ones who would find issues with the OSX installer, though
< gmaxwell>
kallewoof: there was recently another round of massive dusting attacks.
< jarthur>
While some folks are still here, I'm curious if any of you would be against an incremental introduction of unix domain sockets for the RPC API. e.g. release with just server support first, no bitcoin-cli support and no official work around abstract sockets. Server support may be exhumable from wumpus PR #9919.
< jarthur>
bitcoin-cli support is what was held up by proposed upstream libevent work
< gmaxwell>
jonasschnelli: it's hard to test extensively without bitcoin-cli support, I'd worry about it bitrotting.
< gmaxwell>
er jarthur
< gmaxwell>
otherwise getting it in sounds fine to me.
< jarthur>
Yea, there's only so much we'd be testing from the python functional tests.
< sipa>
gmaxwell: if the python test framework uses it, i'd be less concerned
< gmaxwell>
I think thats clearly a requirement.
< gmaxwell>
Still, its not like our tests are comprehensive enough that it's automatically okay.
< jnewbery>
sipa: perhaps not, but lots of people have subscribed to our newsletter who aren't members. We have about 1500 subs, who I expect are mostly quite technical.
< sipa>
jnewbery: true, and i'm a big fan of that work, to be clear
< sipa>
gmaxwell: any type of issue you're worried about in particular w.r.t. unix socket support but excluding bitcoin-cli initially?
< jnewbery>
I didn't interpret your comment to mean that you weren't :)
< gmaxwell>
sipa: e.g. will it crash under concurrent request load, etc. thats something we'd probably find out really fast with bitcoin-cli switched to it.
< jarthur>
On the bright side, having the existing py func tests use a unix socket pretty doable. wumpus had it as an option in his PR.
< jnewbery>
anyway, I don't think it can harm. If there's any advice on how people can better test RCs and provide feedback, I think we'd be happy to push it out to our subscribers.
< jarthur>
My reason for proposing server-only is to make the change small enough to not get sidelined again. I know I don't have enough time to work with libevent team on getting preexisting client connection support rolled in.
< wumpus>
jarthur: yes, I had the functional tests use a UNIX socket, next step would have been to have them interface with P2P over UNIX socket as well but never got that far
< wumpus>
*for RPC
< wumpus>
everyone was too much focused on the cli client support and I'm not confident that I can get the libevent people to adapt anything in that regard so I kind of gave up
< wumpus>
feel free to pick it up though !
< jarthur>
wumpus: thanks. I don't know the client code very well. Is there an easy alternative where we can just have libevent create the socket?
< wumpus>
heh you wish
< wumpus>
I wouldn't have gone though all that trouble if that existed, right :/
< jarthur>
figured
< gmaxwell>
I think it's fine to go without the -cli... just means a somewhat greater amount of testing should be done.
< wumpus>
jarthur: I think the only way to do that would be to implement the entire http protocol (e.g. port over libevhttp), but that's a pretty bad place to be in
< jarthur>
Is it because we do some custom stuff libevhttp doesn't do?
< wumpus>
there's more reasons why libevent's http server isn't really that great for what we're doing (such as the work pool) but there's nothing else that we *can't* do with it
< wumpus>
httpserver.cpp is basically one complex of working-around thread limitations of their http server
< wumpus>
(but that's not relevant to UNIX socket support at least :)
< wumpus>
and there's a more advanced web server that runs on libevent which is commonly recommended: https://github.com/criticalstack/libevhtp ... I don't think that'd work with the client part though, and it's yet another dependency
< wumpus>
I mean, help with the client part, and no idea if they do support UNIX sockets
< wumpus>
I'm...so disasppointed
< wumpus>
(but it's 10000% better than boost asio so...)
< sipa>
haha
< sipa>
gmaxwell: the current way GetStrongRandBytes works is by computing SHA512(state || hw_entropy || os_entropy || openssl_entropy), and then using one half of the output as new state, and one half as random output
< sipa>
for non-strong we can probably replace that with SHA512(state || high-accuracy-timer || some_other_cheap_entropy) or so?
< wumpus>
their server isn't the problem, it allows injecting custom FDs for UNIX sockets fine! it's the client, we'd need another http *client* that supports http over UNIX sockets
< sipa>
wumpus: you may hate this... but given that bitcoin-cli is specific to bitcoind, it probably doesn't need to actually implement full HTTP; just whatever subset is needed for bitcoind
< wumpus>
sipa: yes, that's a good point!
< wumpus>
hadn't really thought of it that way
< sipa>
and if there's an adequate HTTP implementation that does what we need, that certainly feels better than something NIHed
< sipa>
but if there's concerns... i don't expect that the subset of HTTP we need is all that hard to write
< wumpus>
right
< wumpus>
I missed the part where it only needs to work with our own server :)
< jarthur>
wumpus: ahh, was the reason we needed the fd-passing ability because their client didn't even have support for starting a unix socket fd to begin with?
< wumpus>
I guess that makes it quite simple
< wumpus>
jarthur: yes
< wumpus>
jarthur: at the time, at least, if they added it since then that'd be incredible
< wumpus>
(but I doubt it, they've always limited support to the intersection of various OSes, so having a specific unix API is probably even more out of the question)
< jarthur>
Yeah, just checked, they haven't
< wumpus>
too bad
< wumpus>
it's kind of interesting, I mean Tor uses libevent and they certainly support UNIX sockets for many things, for which they use fd injection at the base layer, they just don't happen to use the http server so that's not important
< wumpus>
(nor http client)
< wumpus>
this is just a case of whyyy do you have to make it so difficult for me
< wumpus>
(why does bitcoin use http? I do not know, I guess JSON-RPC is marginally better than rolling yet another custom protocol, though the line-based JSON RPC of c-lightning is *pretty neat*)
< wumpus>
why bother wrapping your JSON RPC requests in all this http stuff when you can just do request\nreply\n
< jarthur>
Perhaps to enable local browser-based experiences?
< jarthur>
Though there are WebSocket transport options for JSON-RPC now
< wumpus>
yea exploitability++
< jarthur>
:)
< promag>
lol
< phantomcircuit>
wumpus, line based json rpc? so stratum?
< bitcoin-git>
[bitcoin] hebasto opened pull request #14577: qt: Cleanup `textInteractionFlags` for `QLabel` (master...20181025-textInteractionFlags) https://github.com/bitcoin/bitcoin/pull/14577
< wumpus>
phantomcircuit: maybe? I've never used stratum, might be similar/the same as c-lightning's protocol
< queip>
<luke-jr> how do users learn about stuff now? <--- make a twitch stream by skimply-dressed lady on configuring RPC ;)
< esotericnonsense>
obviously it's so that you can stick your bitcoin rpc behind nginx and put it on https://myexchange.com/api
< esotericnonsense>
with no auth
< esotericnonsense>
the RPC port won't be exposed as 8332 so it's fine
< wumpus>
which, in some sort of way, makes sense if wallets aren't involved
< * esotericnonsense>
cries
< esotericnonsense>
if it's sufficiently sandboxed... maybe...
< wumpus>
in some ways this is really between a rock and a hard place, most of the information in the API is as public as it gets
< esotericnonsense>
i don't think that's true
< wumpus>
you can put it behind a caching API and it's still fine, this was the idea behind the REST API
< esotericnonsense>
well, I mean, define "most", I guess
< esotericnonsense>
I think someone who looks at my node monitor for long enough would be able to figure out, for example, geographically where it's located
< wumpus>
I mean things like 'what peers am I connected to' can be more or less sensitive?
< wumpus>
I certainly don't mean exposing any information by default, but if someone decides to do it for a public node...
< esotericnonsense>
yes
< esotericnonsense>
sure
< wumpus>
which is what you implied with the nginx thing
< jarthur>
wumpus: re stratum, it's actually implemented on top of JSON-RPC. A lot of the stratum implementations in crypto projects transport JSON-RPC over TCP with line breaks.
< esotericnonsense>
if the actual interface itself is hardened and/or it's running in a jail/VM/whatever with resource constraints (ideally both) I don't see an issue
< esotericnonsense>
i would just assume there are obvious buffer overflows in the rpc interface though
< esotericnonsense>
without really thinking about it too much just because I'm not sure anyone cares (because it shouldn't be public so why care)
< esotericnonsense>
just little things, like what happens if someone sends an rpc request with a string parameter that is 400MB
< esotericnonsense>
lol
< wumpus>
well then it will use 400MB, it's not really interesting
< esotericnonsense>
if not RCE-type overflows then probably a DoS if the server dies
< esotericnonsense>
actually, lol
< wumpus>
you can do that to most websites if you really want
< esotericnonsense>
doesn't rpc just have a shutdown command
< esotericnonsense>
:D
< esotericnonsense>
WRT bitcoin-cli and http/domain sockets/etc
< esotericnonsense>
i'm not suggesting a rewrite, but if it were easier that way, is it necessary to restrict to use of C++? I mean it barely does anything anyway
< sipa>
?
< wumpus>
replace c++ by, waht?
< esotericnonsense>
there's a few config args, the simulated getinfo, help message, and the rest of it is pretty much standard
< esotericnonsense>
anything you needed to to find a non NIH http solution
< wumpus>
I think if you'd rewrite the entire thing in rust overnight a lot of people would be happy xD
< esotericnonsense>
python, say (no idea if python does http over domain sockets)
< esotericnonsense>
what bitcoin-cli?
< wumpus>
noo
< esotericnonsense>
all of bitcoin? :D:D
< gmaxwell>
I think esotericnonsense is suggesting bitcoin-cli get replaced with a python tool?
< queip>
that seems easy... but exactly what for>
< wumpus>
that's trivial
< esotericnonsense>
gmaxwell: i'm not suggesting it but rather responding to wumpus'/sipa message above about bitcoin-cli not needing to be a full http client and inventing some thing
< sipa>
afaik the origin bitcoin-cli code was just some printf statements into a socket
< gmaxwell>
I don't think thats absurd on its face, but: it would probably be a massive slowdown for those users who do processing using bitcoin-cli in scripts, and would suffer the standard "throwaway and rewrite software issues" -- the complexity of existing code is effectively all the accrewed knoweldge built from years and years of use... and that it probably gets right lots of behaviors none of us
< gmaxwell>
ever explicitly realized were ever requirements.
< wumpus>
rewriting bitcoin-cli in python, certainly using the code already in the test framework, would be trivial, but why?
< esotericnonsense>
hm
< esotericnonsense>
i guess I don't hit the right endpoints for the rpc client to be the bottleneck, heh
< wumpus>
I mean it means that all users need to have python installed
< wumpus>
which is the case on linux, and maybe on macosx? but certainly not on windows
< gmaxwell>
esotericnonsense: I mean just forking _python_ for each request is probably going to be a lot slower.
< sipa>
gmaxwell: generally i agree with that... but i also think bitcoin-cli is very simple :)
< gmaxwell>
yea, if we wanted to rewrite anything bitcoin-cli would be it! :)
< jarthur>
esotericnonsense: on Python, aiohttp supports unix sockets out of the box. The popular synchronous libs (httplib, requests) need supplemental libs to enable unix socket usage currently.
< wumpus>
it woult make zero difference for bitcoin-cli wrt performance
< esotericnonsense>
i feel like
< esotericnonsense>
someone using bitcoin-cli in scripts ... argh\
< esotericnonsense>
supporting that feels awful
< sipa>
my preference is just replacing the libevent http code in bitcoin-cli with hand-written HTTP though :)
< gmaxwell>
esotericnonsense: you anti-unix heretic.
< wumpus>
starting python is very fast, certainly if it's already cached in memory, and all work it does is *trivial*
< esotericnonsense>
I mean, with interpretation of output, and so on, the whole point is that json rpc is standard
< wumpus>
python is slow for heavily multi-threaded work but some comand like this?
< gmaxwell>
wumpus: k. I was just guessing, I have no idea if it actually would be meaningfully slower.
< * esotericnonsense>
uses python in production as a http client and gets pretty high concurrent workload
< esotericnonsense>
er, server
< wumpus>
gmaxwell: I mean it compiled everything to byte-code and the interpreter is very small
< esotericnonsense>
it is single threaded though yes
< esotericnonsense>
the overhead of starting an instance does exist, i just wonder about the cases in which that actually matters
< wumpus>
I just don't think it's a relevant concern in this case
< wumpus>
please work on something that actually affects users
< esotericnonsense>
yes
< esotericnonsense>
this is silly, sorry for raising it :P
< esotericnonsense>
jarthur: missed your message, I'm using aiohttp actually, hadn't ever tried it with sockets though.
< gmaxwell>
I don't think it was bad to raise, we might end up back on it if later bitcoin-cli is blocking disabling tcp rpc by default, and libevent still can't handle using domain sockets for the -cli case.
< esotericnonsense>
might have to have a play.
< wumpus>
let's write bitcoin-cli in rust
< gmaxwell>
K.
< * esotericnonsense>
is on page 3 of the rust book after finally getting around to it this morning.
< jarthur>
CC andytoshi
< gmaxwell>
Maybe the rust-bitcoin people have already done it.
< esotericnonsense>
it took me a while to figure out how to add 1 to an integer because x++ doesn't work and that threw me for ages. :D
< wumpus>
there is a rust bitcoinrpc library, but no command-line thing
< esotericnonsense>
jarthur: oh! that's awesome! I don't know if you've had your fingers in any other parts of it but I've been really impressed with aiohttp
< esotericnonsense>
I replaced my old flask backend for my node monitor with aiohttp a week or so ago and it's orders of magnitude faster, (though that is primarily due to using asyncio and not blocking on calls to bitcoind)
< jarthur>
esotericnonsense: thanks! I'm a small-time contributor to it. I'm a big fan, though I tire of all the nested context managers involved in using the client :)
< esotericnonsense>
so cheating, really :P
< wumpus>
which makes sense, why go all the way to compile a rust crate for something then use it from friggin bash xD
< esotericnonsense>
just write bitcoin-cli in bash.
< esotericnonsense>
and powershell.
< esotericnonsense>
done.
< esotericnonsense>
or make windows users use WSL or something. :>
< wumpus>
ahhh shut up or I'm going to rewrite it in risc-v assembly
< sipa>
what's wrong with Visual Basic?
< wumpus>
xD
< midnightmagic>
+1 risc-v assembly
< wumpus>
visual basic 6 was the height of programming language development, after that it's only been a race downward
< sipa>
wumpus: that was the last one i used :p
< wumpus>
me too
< sipa>
after VB6 i learned Perl.
< kallewoof>
gmaxwell: I will work on getting that in place. Though will it actually help in this case? Dusting attacks I mean.
< jarthur>
esotericnonsense: biggest problem I have with working in Python and Core RPC so far is how slow the std lib's JSON parsing/creating is, just because it's written in Python itself. I need to see if electrumx will be cool with switching to the raw REST APIs for block retrieval.
< kallewoof>
gmaxwell: It seems like a more helpful thing to push for is #13756
< esotericnonsense>
jarthur: aaargh. you just reminded me.
< esotericnonsense>
i spent a while hacking together some string stuff to optimize a json call the other day.
< esotericnonsense>
i forgot to uncomment 'import ujson as json'.
< esotericnonsense>
-_-
< jarthur>
ahh, yea, that'll help :D
< esotericnonsense>
can't be bothered to switch it back now until I need to change it. lol.
< wumpus>
midnightmagic: so for all other platforms we'll have to ship qemu to emulate it
< esotericnonsense>
wumpus: no just write the bits of qemu you need to emulate bitcoin-cli on riscv.
< esotericnonsense>
obviously.
< wumpus>
I have a risc-v emulator in rust ?
< * esotericnonsense>
goes to get a beer.
< wumpus>
:')
< wumpus>
sipa: I've also used perl for a little while, don't remember anything though
< * esotericnonsense>
can't really remember his route
< esotericnonsense>
i think qbasic, c, ???, python, then just small amounts of arbitrary languages?
< wumpus>
something like MSX basic -> Z80 asm -> gwbasic/qbasic -> PASCAL -> C -> perl -> C++ -> python -> rust ... though I don't exactly remember anymore, and there's some languages like haskell that I tried but never got very far with
< aj>
oh, actually would've been quickbasic not qbasic
< sipa>
oh, there's a small amount PHP somewhere
< jcorgan>
after Forth it all went downhill
< wumpus>
oh I forgot about PHP as well, that was between perl and C++ I guess
< sipa>
between java and haskell for me
< wumpus>
aj: Ada is neat, I think it's a nice evolution of pascal, too bad it's mostly only used for us military stuff
< aj>
wumpus: the precondition/postcondition stuff was pretty nifty. i remember it being a gratuitious pain to program with though, for what felt like syntactic reasons. might've been weird manual array management that would be completely laughable these days, don't remember
< aj>
wumpus: i was never a fan of pascal though *shrug*
< esotericnonsense>
i think I used PHP enough to think 'f this'
< wumpus>
aj: I *loved* pascal, it was only because C seemed to be the wave of the future that I learned it and moved to it
< wumpus>
that's a loong time ago though, I don't think I could do much with it these days
< wumpus>
but things like the module system, bounded arrays, actually enforcede enumerations, it seemed to be ahead of C in many things
< wumpus>
C was pretty nice as a low level, platform independent assembler replacement, but then they started to actually enforce things like undefined behavior, and platform-dependent behavior became important, and heck all of today's pain
< aj>
wumpus: i guess bounded arrays, enforced enumerations are all strong-typing things where you'd go to haskell (or maybe rust?) these days
< wumpus>
aj: it's why I like rust I think, I also *like* Haskell but cannot do anything practical with it
< sipa>
i absolutely loved haskell as a language, and in many ways wished mainstream languages were more similar to it
< sipa>
except in practice the type of programs i like to write are ones where performance and predictable resource usage matter... exactly what it abstracts away
< jcorgan>
the older i get, the more languages i learn, and the fewer i use
< jcorgan>
i'm pretty much down to python and c++ these days
< aj>
sipa: haskell, but with performance specified as part of the type ;)
< sipa>
not to say that you can't write performant code in haskell, but it feels like you need to break the otherwise awesome abstraction it provides in order to do so
< sipa>
i need to learn rust
< jcorgan>
the last language to really interest me was Go, but that was self-limiting
< esotericnonsense>
I like go
< esotericnonsense>
i usually just run into a wall of sort of... OK, this is an interesting language, but does anything I do actually matter enough to optimise for language
< esotericnonsense>
the answer is no, so I just end up reaching for python
< jcorgan>
i liked Go's built-in concurrency and message passing
< * sipa>
discovers the -j option to test_runner.py
< wumpus>
sipa: +1 on Haskell, it's nice and pure but also I really like to understand how things map to the mechanical reality how things run on CPUs, and that's hard to grasp for me
< * esotericnonsense>
has opened the rust tutorial again.
< gmaxwell>
kallewoof: the dusting party was paying people that had existing outputs.
< esotericnonsense>
I killed my vfio VM and can't be arsed to reboot the whole box to get it to work. heh
< gmaxwell>
kallewoof: so if we automatically would spend that dust it would help and make the attacks less interesting.
< sipa>
(maybe the language discussion is getting a bit off topic here; sorry for continuing it myself)
< gmaxwell>
kallewoof: the forced avoidreuse is interesting too by my opinion is that off by default behavior is almost irrelevant from a privacy perspective. We should still do it, as part of a prinicipled commitment to privacy, but in practice if it's not on by default it's mostly not helping people.
< wumpus>
yes, sorry, I don't really want to say much about language, except taht I increasingly dislike c++ and like rust
< gmaxwell>
I think that probably it'll only be aggregatable signatures that really get us to where we need to be on that.
< kallewoof>
gmaxwell: If I dust your bc1qgmax you will never use that "group" unless you turn on -avoidpartialspends, because using it will always give a higher fee.
< kallewoof>
gmaxwell: with the "do if it gives same fee" feature, I mean.
< gmaxwell>
kallewoof: not necessarily, because the alternative spend could have just as many but different inputs.
< kallewoof>
But if it's dust, will that ever happen?
< gmaxwell>
kallewoof: sure, you can miss by any amount, if your payment plus fees ends up being 5 satoshi short, then you're going to need another input and a dust one might be perfectly reasonsable.
< kallewoof>
Sounds unusual but sure! :) I'll work on getting that back in. I am not sure the approach I had was reasonable though (doing the entire coin selection twice and comparing), esp for bigger wallets
< gmaxwell>
kallewoof: why do you think it's unusual? I expect the amount missed in any coin selection to be a small value with a peak near zero.
< gmaxwell>
And importantly, this really can be always on for everyone with effectively no compromise (beyond the development effort)
< gmaxwell>
It also can be generalized to a form where you'll pay extra for it by some configurable small amount, but not an unbounded amount extra... which is I suspect what joe-user probably wants.
< kallewoof>
Oh, that sounds pretty useful, yeah.
< kallewoof>
The only compromise is that coin selection takes 2x the time. For big wallets I hear rumors that this may be a big deal actually
< kallewoof>
I should probably make a big wallet and see for myself.
< gmaxwell>
I don't think it would on those wallets.
< gmaxwell>
Also if you hear rumors like that you should invite people to report them, there is a lot of rumor that turns out to be spurrious.
< kallewoof>
I think it was rumors that came from some exchange. I don't know if they're comfortable revealing too much about their set up.
< gmaxwell>
Considering that we used to sign in the coinselection loop and are now well over 100x faster for many input spends, I suspect we have headroom. :)
< kallewoof>
Should be easy to test though. Just make a wallet with a ton of UTXOs
< kallewoof>
Oh.. wow.
< gmaxwell>
Right but the rumors aren't useful reports-- because they can't distinguish 'slow' being from things like signing in-loop for many input spends vs basic behavior. If businesses want to get behaior out of us, they need to behave like professionals.
< gmaxwell>
kallewoof: also we could just have an option to turn it off if we're concerned... and the few parties that care could flip it.
< wumpus>
yes, the problem has always been that the big users neither contribute actually useful reports, nor developers that optimize what is important for them
< kallewoof>
gmaxwell: sdaftuar spoke against the feature in the original PR here: https://github.com/bitcoin/bitcoin/pull/12257#discussion_r204168100 ("I'm not sure that -avoidpartialspends works very well on wallets that have substantial address reuse, in particular I think you can devolve into cases where you'd produce giant transactions that take forever to sign and would never pass policy (or consensus) limits, in extreme cases.")
< kallewoof>
The concern was partially mitigated by putting a cap on the UTXOs in a single group so maybe sdaftuar is cool with the idea now.
< wumpus>
oh it's slow if you have tens of thousands of utxos, yea... that's awesome, no one that actually partakes in development uses it with that
< gmaxwell>
wumpus: it shouldn't be.
< wumpus>
of course it shouldn't
< gmaxwell>
kallewoof: re coinselection speed, another thing that used to make it slow is having a lot of unconfirmed outputs... we got 1000x fold speedups due to fixing it so that IsMine didn't have quasi-factorial complexity.
< * kallewoof>
looks up quasi-factorial
< echeveria>
from my experience people making a lot of these sort of claims don’t actually use the software at all, so keep that in mind too. I’ve had people sit and insist to me that the bitcoin wallet has all sorts of stupid behaviour I full well know it doesn’t.
< gmaxwell>
I think we need to draw a line in the sand. If these large commercial players can't behave professionally enough to actually report their concerns (much less contribute effort to fix them and/or add test cases) we should ignore them... not fall into some rumor based development.
< wumpus>
but I couldn't much be concerned about companies that make millions from using the software but contribute nothing back
< gmaxwell>
Right.
< kallewoof>
Makes sense to me
< echeveria>
some of this comes from parties like blockchain.info who have frequently gone around conferences repeating stories of how they tried so hard to contribute to bitcoin core and were rejected for their efforts.
< gmaxwell>
(I very much want their input for sure, but if they don't provide it.. they don't provide it. Unfortunately, my expirence with many bitcoin companies is that they have executives that have never actually used software in a commercial context before, much less open source... and they treat it like the android play store... an app doesn't do what you want, you swap out for another one).
< kallewoof>
heh
< echeveria>
now if that’s true or not I don’t know, but I’ve not seen that sort of behaviour. there’s some contributes in the source tree from people I wouldn’t want to be in the same room with- because they’re evaluated on their merit rather than where they are from.
< gmaxwell>
hopefully optech will do a useful job in getting more input, but otherwise we should just do the best we can. :)
< gmaxwell>
echeveria: I personally liked the one company going around telling everyone that bitcoin core constantly crashed, and then got shut up by a developer offering them a sizable bet that they couldn't report any crash...
< sipa>
in particular about coin selection, one question that came out of optech was making coin selection code more accessible... which sounds to me like they're not using the built-in wallet, and thus won't ever observe the coin selection logic