< tryphe>
anyone know if -addnode connections count towards the total of maxconnections? no, right?
< gmaxwell>
tryphe: max connections, yes, but not your outbound connections
< gmaxwell>
Why?
< wumpus>
yes, maxconnections is a hard limit (to be able to limit memory and file descriptor usage), there should be no way to exceed it
< tryphe>
gmaxwell, i'm about to open a PR that addresses a bug and has some logic fixes but i'm unsure about a small part of it, in terms of the "minimum" number of file handles we should request in terms of addnode/outbound/feeler connections. MAX_ADDNODE_CONNECTIONS + MAX_OUTBOUND_CONNECTIONS + feelers = 8 + 8 + 1, but i'm not sure if that should be the minimum for -maxconnections. and if it shouldn't be, there's no way to lower those constants at
< tryphe>
the moment.
< gmaxwell>
tryphe: okay, reviewing the logic: reducing maxconnections can (if you set it low enough) reduce the outbound connections. But it doesn't reduce the addnode connections, because how would it decide how to split up the reduction. Addnode connections are handled seperately for the FD limits.
< gmaxwell>
tryphe: there probably doesn't need to be anything to lower the max addnodes: if you want fewer, add less addnodes.
< tryphe>
gmaxwell, i just meant in determining a "soft cap" for the theoretical minimum number of handles that could be used in the normal daemon, 8+8+1 seems to be the answer there. ie. it's unknown at runtime whether the user will need it, but they might.
< wumpus>
interesting, so maxconnections isn't really maxconnections
< gmaxwell>
tryphe: if maxconnections is lowered one of those 8s will go down.
< tryphe>
gmaxwell, and in this context i mean only handles for sockets
< wumpus>
I think it used to be different, but no sure...
< tryphe>
it got changed about a year ago with the new select() code i believe
< gmaxwell>
wumpus: right, I'd forgotten about that. It's because if maxconnections is set lower than 16 how do would decide how to share with addnode? and addnodes can be limited by not using them.
< gmaxwell>
wumpus: it was probably done by the change to make addnodes not use up the outbound connection slots.
< gmaxwell>
(which made addnode unreliable, since outbounds would eat up all its slots)
< wumpus>
gmaxwell: I would have expected addnode to fail at some point if it requests adding connections and there are no slots available, but I agree
< gmaxwell>
it wasn't an unintentional change, I just didn't remember it.
< wumpus>
addnode is always a manual action so if you run out of resources that way, it's your own fault
< tryphe>
i'm not using addnode, i'm just asking for the bug's sake (you'll see in a second, hehe)
< tryphe>
it probably needs some fixing
< gmaxwell>
wumpus: right but the problem that drove splitting them was this: You add an addnode. Your node comes it it uses it. Great. Then days/weeks/month later your addnode connection disconnects... now it'll never reestablish it because an automatic outbound took the 'free' socket. So the alternatives were to make addnode disconnect outbounds (which is kinda ugly since the addnode reconnect might
< gmaxwell>
fail), or -- not share a limit.
< gmaxwell>
tryphe: sure, we've been cheating for a while with the "MIN_CORE_FILEDESCRIPTORS"
< tryphe>
if you set ulimit -n 150, you get this message: Warning: Reducing -maxconnections from 125 to -8, because of system limitations.
< tryphe>
:D
< wumpus>
gmaxwell: yes it gets quite complex in that case, could reserve the addnode slots and not make normal outbound connections in them (so there's nothing to disconnect), but, I think this is ok
< gmaxwell>
wumpus: yep, though then we'd have an odd behavior that setting maxconnections < 8 would cause you to make no connections. That might not be so bad except it would be a surprise for people with maxconnections already set to some low value.
< bitcoin-git>
[bitcoin] tryphe opened pull request #16003: [init] an incorrect amount of file descriptors is requested, and a different amount is also asserted (master...fd-limits-3) https://github.com/bitcoin/bitcoin/pull/16003
< wumpus>
tryphe: whoops it should error out in that case I think
< gmaxwell>
tryphe: min would be 150 + 8 + 1 + 1 + 1 I think the MAX_OUTBOUND_CONNECTIONS gets reduced.
< gmaxwell>
tryphe: sounds like you're on the right path to me, though
< tryphe>
gmaxwell, hmm, but i wonder how the net code regards those consts. if i have -maxconnections=8 i don't know how many -addnode connections i -should- be able to make, for example
< tryphe>
versus other connections i mean
< gmaxwell>
tryphe: 8. it'll make up to 8 addnode connections regardless of what maxconnections is set to.
< tryphe>
ahh i see, thanks
< tryphe>
i figured that's how the feeler was as well
< gmaxwell>
tryphe: wumpus and my initial answer were incorrect, addnodes don't charge against maxconnections.
< gmaxwell>
Nor does the feeler IIRC though I haven't gone and checked again (feeler is short lived-- matters for FD accounting reasons)
< tryphe>
okay, so i should probably exclude that from nUserMinConnections, but still include it in nFDMin
< tryphe>
i think?
< gmaxwell>
no.
< tryphe>
oh, nevermind
< gmaxwell>
if the user runs with a silly ulimit that only lets them make 0 outbounds, I suppose thats their own problem!
< gmaxwell>
(though obviously it should be logging that fact.)
< tryphe>
yeah, but there are systems like mac os where the requested amount comes close to the actual system limit, unlike the 1024 default in most Linux distros
< mryandao>
is there even a use-case to apply ulimit changes before running an application?
< tryphe>
mryandao, it was simply the only thing i could find to show that there's an issue :p
< gmaxwell>
tryphe: on macos it increases it.
< gmaxwell>
Erroring out would not be a good thing to do by default on any reasonably popular system.
< wumpus>
mryandao: no, but some OSes do that automatically, I remember OpenBSD used to have very low FD limits for default users, don't know if this is stil true
< gmaxwell>
right many of the BSDs set soft limits low (actually inhereted behavior from older unixes), but applications can increase them.
< wumpus>
gmaxwell: I think it should error out only if it's not able to function normally, e.g. even reserve the file descriptors for the databases and default # outgoing connections
< wumpus>
if your 'available FD count' ends up in the negative I'd say that is true
< gmaxwell>
So this stuff about erroring out is only if we've run into the hardlimit (can't increase more) and still don't have enough to work.
< wumpus>
much better to fail initially than later on when it's unable to open a file
< mryandao>
its unusual for an application to be able to increase ulimit when the user has pre-define a lower ulimit before executing the application.
< wumpus>
gmaxwell:yes, that's what I mean
< mryandao>
what if an application decides to set to unlimited?
< gmaxwell>
mryandao: it's _really_ common for applications to increase the soft limits. Soft limits everwhere have to be 1024 or less to prevent breaking select.
< gmaxwell>
mryandao: so anything that doesn't have that problem increases it, your browser does, openoffice does, etc.
< gmaxwell>
If you want to prevent that you lower the hard limit.
< tryphe>
mryandao, the daemon can't increase the limit if you have it specifically set, though
< tryphe>
hence 125 max connections is dynamic
< gmaxwell>
I'm not aware of any system that has a particularly low default hard limit.
< tryphe>
it can lower 125 to as low as 17 (with my PR), it doesn't return an error, but i feel like people might still be dumb and try to set it lower and expect it not to make those connections
< gmaxwell>
I don't think setting maxconnections higher than your system ulimit max count can support should result in a failure to start, it should do as it does now: log that its reducing your max connections.
< gmaxwell>
It's debatable if it should refuse to start if it can't even get enough for 8 connections, I could go either way on that.
< gmaxwell>
on one hand, its normal operation, on the other hand you might be noconnect and not making any connections at ll.
< gmaxwell>
all*
< gmaxwell>
sounds like wumpus prefers to at least preserve the ability to make 8 connections (I supose unless you've set maxconnections lower than even that)
< tryphe>
yeah, presumably you'd be able to run -connect=0 if you want no connections or something, as i think -maxconnections was intended to only increase connections and not specify what types of connections are to be used.
< tryphe>
like if i have -maxconnections=1, does the feeler socket use the connection? it would be nice to mention something like that in the docs probably
< tryphe>
i mean mention that a value below is invalid, or something
< tryphe>
below x*
< gmaxwell>
tryphe: no maxconnections is not intended to only increase connections.
< gmaxwell>
In fact historically it could not increase connections.
< tryphe>
gmaxwell, i guess i never ran with -maxconnections=0
< gmaxwell>
tryphe: feeler and addnode do not eat into maxconnections count, I believe. (well I know for addnode, pretty sure for feeler)
< sipa>
it's a way to set resource limitations
< gmaxwell>
max connections is the limit on automatic and inbound connections.
< sipa>
running with -maxconnections=20 was fairly common advice on low-memory systems (at a tike when we had huge per-peer network buffers; it's much less impactful now)
< bitcoin-git>
[bitcoin] nalinbhardwaj opened pull request #16006: rpc: use walletrbf as default setting in walletcreatefundedpsbt (master...walletcreatefundedpsbt) https://github.com/bitcoin/bitcoin/pull/16006
< tryphe>
gmaxwell/wumpus: i smoothed some things out better, but still not sure if that feeler connection should be added to the initial handle minimum. let me know what you think.
< tryphe>
handle/descriptor*
< bitcoin-git>
[bitcoin] nalinbhardwaj closed pull request #16006: rpc: use walletrbf as default setting in walletcreatefundedpsbt (master...walletcreatefundedpsbt) https://github.com/bitcoin/bitcoin/pull/16006
< gmaxwell>
tryphe: I think they should, you can't deactivate the feeler.
< meshcollider>
But both are his PRs and he already has one
< jnewbery>
I think 15761 will be removed as a requirement based on the IRC meeting a few weeks ago
< gmaxwell>
sipa: an upper bound on the cost needs to be known before signing starts.
< achow101>
hi
< achow101>
meshcollider: 15761 isn't a requirement for descriptor wallets anymore
< achow101>
15741 isn't necessarily a requirement, but it makes things faster
< meshcollider>
achow101: ok, that's good
< sipa>
we probably need something like 15741 anyway
< meshcollider>
But not necessarily on high priority atm, I guess we will leave it as-is
< meshcollider>
#topic signing scripts were different satisfactions may have different costs (sipa)
< sipa>
gmaxwell: yes, the easiest approach is always assuming the worst case
< sipa>
this is in the context of things like miniscript or the taproot proposal i recently published
< gmaxwell>
sipa: that isn't quite what I meant, like if you're going to spend via branch X, you have to know that in advance if you want to use lower weight for fee purposes.
< gmaxwell>
so I think PSBT may need an extension for that.
< sipa>
right, but plugging that into fee estimation and coin selection seems nontrivial
< gmaxwell>
I think its trivial once you assume you have a way of knowing the "weight bound" for each input you're going to use... which itself is only triial if you always assume the worst case branch.
< sipa>
gmaxwell: hmm, i guess if we can come up with something sufficiently generic to put in PSBT (something that restricts certain options or so?), it can probably go in the same form into descriptor records
< gmaxwell>
right.
< gmaxwell>
my thought is that a descriptor should be subsettable.
< gmaxwell>
Like if a script is A or B, there should exist a descriptor that maps to the same spk but only lets you spend via A
< sipa>
that's an interesting idea, putting it in the descriptor itself
< gmaxwell>
in the context of taproot, that descriptor might not even reveal the content of B.
< gmaxwell>
Descriptor-slice.
< sipa>
let's call it a subscriptor
< sipa>
:p
< gmaxwell>
oohhhh
< meshcollider>
lol
< gmaxwell>
Right, so basically you make the cost analysis use the worst case, but use of a subscriptor can lower the worst case.
< achow101>
so if used with taproot, you would have the hash of the other branch indicating that that other branch won't be used
< gmaxwell>
right. something like that. I think you should be also able to include the data but indicate it won't be used.
< sipa>
i guess there could be an unavailable(...) syntax element in descriptors, which for output calculation is identical to ..., but assumes the key/path/... subexpression isn't available for signing
< gmaxwell>
(for a lot of applications you'll want to know what it is)
< sipa>
(or something more syntax sugarry)
< sipa>
i like this
< achow101>
but does such a subscriptor need to be included in a psbt?
< sipa>
i think a subscriptor could just result in certain information not being put in a PSBT
< sipa>
like certain branches of a merkle tree (assuming a taproot psbt extension) would just be left out if they're known to be unavailable (or just unknown)
< achow101>
right
< sipa>
thanks, i don't think this much more discussion right now
< meshcollider>
Any other topics then?
< meshcollider>
Is there anything else related to the Taproot/schnorr proposals that anyone wants to discuss here?
< gmaxwell>
sipa: will you be doing a miniscript that targets taproot?
< gmaxwell>
(like a compiler that takes the current input and outputs taproot scripts)
< sipa>
gmaxwell: obviously :)
< gmaxwell>
if you were planning to anyways, it might help discussion around taproot because you could compile example scripts both ways and show how their minimum and worst case spending costs change.
< sipa>
meshcollider: i think most wallet discussions related to that are for later
< gmaxwell>
manually constructing examples is always a bummer (and easy to get wrong)
< sipa>
(afk now, will be back in an hour or so)
< meshcollider>
Ok let's end things here then, thanks everyone :)
< meshcollider>
#endmeeting
< lightningbot>
Meeting ended Fri May 10 19:25:50 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< instagibbs>
was going to ask "when miniscript in Core"
< instagibbs>
motivating use-cases are probably required, but above conversation answers that one way
< gmaxwell>
instagibbs: well, maybe it makes sense to do miniscript with taproot and not bother without.
< instagibbs>
yep
< gmaxwell>
certantly, things like taproot need miniscript in the sense that their ability will be wasted if we don't make it easier to make complex scripts.
< sipa>
jonasschnelli: now that you have benchmarks for your poly1305/chacha20 implementation... would it be much work to also implement the "standard" openssh-like construction?
< sipa>
it'd be good to have numbers to justify the choice for our own modification