< BlueMatt>
gmaxwell: the fd_set 1024 limit stuff shouldn't cause that kind of problem, ldb shouldnt use almost any fds on 64 bit and should use a low-ish count on 32 bit, no?
< BlueMatt>
I thought that's where we landed (as mmap doesnt need to use an fd)
< sipa>
ossifrage: are you on a 32-bit system?
< BlueMatt>
fwiw I think I have a branch with a rdwr lock implemented somewhere, though I never ended up using it
< sipa>
can you implement a shared lock without support from underlying primitives?
< sipa>
seems yes, after short googling
< BlueMatt>
sipa: sure? with atomics and condvars, ofc
< sipa>
yeah
< sipa>
with just locks and condition variables you can implement any synchronization primitive, i believe
< gmaxwell>
ossifrage: why did you say above that ldb was using lots of FDs?
< ossifrage>
That was from the output of lsof and some counting
< BlueMatt>
that seems...surprising, given its supposed to do mmap and then close the fd
< sipa>
i have a number of chainstate files open by bitcoind as well - most are mmaped, but not all
< BlueMatt>
so it may still show up in lsof but not use an fd?
< ossifrage>
I was counting file descriptors not maps
< sipa>
i have 30 chainstate files open with an FD, and 999 with mmap
< ossifrage>
The node has been up for >30 days if that makes any difference
< gmaxwell>
come on, why can't we just take the not-many-line-change to use poll? I know libevent future ra ra ra... but we have held off this simple fix for years. :(
< sipa>
gmaxwell: we totally should
< ossifrage>
I'm willing to test :-)
< BlueMatt>
its super trivial to write
< BlueMatt>
would take me longer to dig it up than someone rewrite it
< gmaxwell>
I could dig up an old copy, but I know that phantomcircuit and BlueMatt run nodes with it.
< sipa>
but i've also basically never encountered anyone actually seeing the "FD above 1024" error resulting in actually closed connections
< sipa>
so i wonder what is special about ossifrage's setup
< sipa>
or maybe just nobody ever paid attention to it
< gmaxwell>
more fragmentation of databases? You do see a bunch of files open.
< gmaxwell>
It also can be caused by RPC connections using up FDs.
< BlueMatt>
I dont bother running mine with it anymore, and regularly have 500+ connections
< BlueMatt>
at least that's usually when one asshat makes 100 connections, but usually that asshat is me
< sipa>
having hundreds of files would certainly explain fd<1024 shortage
< BlueMatt>
and I have a ton of scripts that poll rpc regularly, but not constant, sooo
< ossifrage>
I have txindex and sadly my bitcoin data is on a btrfs filesystem (a mistake I won't make again)
< BlueMatt>
both of those are also true on my seednodes
< gmaxwell>
I think we have had other complaints about fd shortage... but I think we were chalking them up to rpc.
< BlueMatt>
I *know* there are issues with rpc
< ossifrage>
The only reason I noticed a problem was I was dropping a ton of connection due to select()
< gmaxwell>
ossifrage: and the problem remained after restarting the node?
< ossifrage>
gmaxwell, I haven't restarted the node
< gmaxwell>
I wonder if this is a high uptime + txindex + only guy in that config who is watching the logs problem?
< gmaxwell>
it would be nice to better understand why leveldb is leaving the files open... but ... switching to poll eliminates all these problems.
< ossifrage>
both txindex and chainstate are gobbling up file descriptors (let me count the maps)
< BlueMatt>
ossifrage: are you sure its using an fd, or just mmap, cause mmap *shouldnt* at least for ldb
< sipa>
BlueMatt: i confirm that lsof shows both fd-ful opened files and mmapped ones
< BlueMatt>
that...sucks
< ossifrage>
685 maps of chainstate/*.ldb and 268 maps of txindex (odd)
< bitcoin-git>
bitcoin/master 9994d01 Jesse Cohen: Add Unit Test for SingleThreadedSchedulerClient...
< bitcoin-git>
bitcoin/master b296b42 Jesse Cohen: Update documentation for SingleThreadedSchedulerClient() to specify the memory model
< bitcoin-git>
bitcoin/master cbeaa91 Jesse Cohen: Update ValidationInterface() documentation to explicitly specify threading and memory model
< sipa>
ossifrage: yes
< BlueMatt>
MarcoFalke: wat
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #13247: Add tests to SingleThreadedSchedulerClient() and document the memory model (master...scheduler-tests) https://github.com/bitcoin/bitcoin/pull/13247
< BlueMatt>
oh, it did get rewritten to be r-a, nvm
< gmaxwell>
sipa: also even if the issue didn't exist on x86_64 (though it seems it does), we'd still have it on 32 bit.
< sipa>
how do i find the uptime of by bitcoind?
< gmaxwell>
there is a starttime in node info or something?
< ken2812221>
uptime?
< ossifrage>
Oh, in my case it was bitcoin-qt not bitcoind
< ken2812221>
rpc
< sipa>
ah, i tried getnodeinfo and getuptime
< sipa>
seems it's just 'uptime'
< sipa>
10 days, apparently
< ossifrage>
32 days, currently with ~160 connections (which seems to be the most I can get, and I think it has been shrinking as more fds are used)
< ossifrage>
I was going to build a new version, is it useful to reduce the max fd open count for leveldb?
< sipa>
it may impact performance
< sipa>
gmaxwell: on 32-bit systems we limit the max open files to 64
< ossifrage>
I've mmapped >10k pgm files on this box at one point, not sure why leveldb wouldn't just mmap all of the ldb files?
< gmaxwell>
ossifrage: oh you've increased your max connections over default.
< gmaxwell>
so that might be one reason you're seeing this and we are not getting other reports.
< ossifrage>
gmaxwell, yeah I have a gbit connection, figured I might as well get the most out of the blood money I pay verizon
< ossifrage>
But if leveldb where to use 1024 fds, there would be nothing left for sockets
< gmaxwell>
interesting to me that you're actually ending up with that many peers.
< ossifrage>
gmaxwell, it takes a while, but the connection count slowly increases over time
< ossifrage>
Before I had a UPS on my ONT, I'd change IPs ever power failure and then it would take a long time before the connection count would go back up (with the new address)
< sipa>
i have 148 connections
< gmaxwell>
cool.
< gmaxwell>
some months back, on my long running nodes I was unable to break 125. must be more inbound right now.
< ossifrage>
The txindex has been useful a few (rare) times, but just turning that off would delay the problem
< gmaxwell>
(well, or spies, mine blocked spies really agressively)
< ossifrage>
that is ~160 connections (with your block list)
< gmaxwell>
yea, though I haven't updated the blocklist for a while
< sipa>
ossifrage: txindex on or off wouldn't affect the behaviour w.r.t the chainstate ldb files
< gmaxwell>
(right now I hav no nodes with public inbound)
< gmaxwell>
sipa: yes but he is also losing a bunch of fds to txindex.
< sipa>
and your number for those is on itself pretty high
< ossifrage>
sipa, yeah but it was the chainstate + txindex that pushed the fd count >1024
< gmaxwell>
Poll is also good because its faster.
< ossifrage>
I'd happily test a patch :-)
< sipa>
do we require Vista or later?
< sipa>
because windows introduced a poll equivalent in Vista
< achow101>
yes, xp support was removed a few versions ago
< gmaxwell>
I think we don't formally support older versions but they still work.
< gmaxwell>
?
< sipa>
xp certainly doesn't work anymore
< gmaxwell>
Also at least in theory we might work on some other platform that doesn't have poll. We could try switching to only poll and see if someone complains.
< achow101>
"Microsoft ended support for Windows XP on April 8th, 2014, No attempt is made to prevent installing or running the software on Windows XP, you can still do so at your own risk but be aware that there are known instabilities and issues. Please do not report issues about Windows XP to the issue tracker." <-- from 0.14 release notes
< ossifrage>
I use XP to do my taxes, amazingly the tax software works on it
< gmaxwell>
sipa: in windows select doesn't have the 1024 fd limit thing
< gmaxwell>
it's implemented as a linked list or something.
< luke-jr>
array of fd numbers IIRC
< luke-jr>
and it does have a limit
< luke-jr>
just not for the fd numbers themselves
< luke-jr>
defaults to 64
< ossifrage>
I thought on linux you could call select() with larger fdsets and it would work, but the libc fd_set is a fixed size?
< ossifrage>
But it is not exactly efficient, especially with sparce sets
< luke-jr>
you're supposed to be able to #define FD_SETSIZE before including stuff, to get more, but last I checked that was broken in glibc
< ossifrage>
I've used epoll() for so long, using select() just makes me sad
< gmaxwell>
ossifrage: indeed, but unfortunately BSDs and linux solved "life beyond poll" differently.
< ossifrage>
gmaxwell, yeah that was never a concern for the stuff I was writing
< ossifrage>
Sure it's portable, you can port it to any linux you like
< gmaxwell>
we manage few enough connections that poll is fine anyways.
< phantomcircuit>
gmaxwell, think you're looking at that graph wrong
< phantomcircuit>
smaller is better
< phantomcircuit>
or did you mean that poll() is the same as select() ?
< phantomcircuit>
select and poll do basically the same thing just with a much better api for poll
< phantomcircuit>
both pass the entire list to the kernel in every call
< ossifrage>
epoll() was a big win for high connection count, low traffic
< Dave18>
With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/
< Dave18>
I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/
< Dave18>
A fascinating blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/
< ken2812221>
spam again
< phantomcircuit>
gmaxwell, epoll and kqueue are virtually identical
< phantomcircuit>
it's really silly
< midnightmagic>
I think there's something even more different in NetBSD too.. kevent()? kfilter_register()? I forget now.
< gmaxwell>
phantomcircuit: yes, I was saying I thought poll was somewhat faster, but apparently not.
< gmaxwell>
phantomcircuit: go PR that poll patch.
< midnightmagic>
There are faster things than poll if you use whatever they provide natively.
< gmaxwell>
yes, but faster is not generally our issue, max connections = 100, or at most a few hundred.
< midnightmagic>
(also no limitations a la select()'s irritating problems) -- and with a usage of the native event'ing things get very scaleable. But.. not like anyone but somoene like me is going to run a larger-scale system with NetBSD anyway.
< phantomcircuit>
midnightmagic, the issue is that epoll and kqeueue and whatever windows uses are all platform specific
< phantomcircuit>
there's some work being done to move to libevent but that's not done
< phantomcircuit>
the poll() thing is pretty trivial iirc
< phantomcircuit>
80% solution for 10% the effort
< gmaxwell>
phantomcircuit: open the PR!
< gmaxwell>
I know you have had a patch.
< phantomcircuit>
it's like 3 years old now but should be trivial to rewrite
< midnightmagic>
using things like kevent natively isn't hard, it just needs to be clean and people on those platforms will look after it.
< gmaxwell>
true but not obviously of any real value.
< phantomcircuit>
midnightmagic, under thousands of fds it doesn't much matter
< gmaxwell>
phantomcircuit: PR PR PR
< phantomcircuit>
gmaxwell, yeah yeah
< phantomcircuit>
gmaxwell, i remember
< phantomcircuit>
windows is WSAPoll not poll
< phantomcircuit>
and all the types are insane
< phantomcircuit>
like it's virtually identical
< phantomcircuit>
but not
< phantomcircuit>
but i guess that codes already full of hacks for that anyways
< gmaxwell>
phantomcircuit: windows can keep using select, it doesn't hae the same limit.
< sipa>
yes, but its fdset implementation is a linked list with horrendous performance
< gmaxwell>
so? I mean, we only need it to support max-connections. and it already does.
< gmaxwell>
or is it so bad that its noticably slow even for 125 connections?
< sipa>
probably not
< luke-jr>
last time I checked, the only better alternative Windows had required significantly re-architecturing most programs to use it
< luke-jr>
something along the lines of async IO, rather than write-ready notification/checking
< ossifrage>
My computer shat itself, but I found the leveldb mmap limit and bumped it from 1000 to 4096, hopefully that will address my problem
< wumpus>
luke-jr: yes let's definitely not do that, last thing we want to maintain is complex specificially for windows rearchitected network code in the repository
< kallewoof>
Are there cases where the rev file for a previous blkXXX file are modified? Is this something that happens often? I assume it only happens at reorgs, in which case it should be very seldom except at transition to XXX+1
< kallewoof>
s/are modified/is modified/
< wumpus>
kallewoof: no, that never happens
< wumpus>
kallewoof: only the last blkXXXXX file is written too, other files are only potentially deleted (pruning)
< wumpus>
in case of a reorg old blocks will not actually be overwritten
< kallewoof>
wumpus: really? what if a 2-block reorg happens right after a new blk file was created containing a single block?
< kallewoof>
the rev file was for reorgs, i thought
< wumpus>
same for rev files, as far as I know, the data for rev-ing the old blocks will stay there
< wumpus>
it just won't be referenced by the active chain anymore
< kallewoof>
wumpus: Huh, okay. Well, that's good news for masterdatadir PR then
< wumpus>
yes
< wumpus>
that principle works, I've been using it for a long time with an external script
< kallewoof>
wumpus: Wait, ldb files are readonly too? Right now I am copying the chainstate over (~4 gb)
< kallewoof>
wumpus: Though I can't really use the same approach there... wonder if it would be useful to check for hard linking capabilities and using them if found...
< kallewoof>
So, about lint-locale-dependence.sh, which by the way has a list of violations about as long as the linter itself, complains about a bunch of functions because they are locale dependent. But there is no alternative (fix). If you need e.g. std::strtoull() you need to add to the list of violations in the linter. Is this even useful at all, when there are no non-locale dependent alternatives you can switch to?
< gribble>
https://github.com/bitcoin/bitcoin/issues/13676 | Explain that mempool memory is added to -dbcache during IBD by Sjors · Pull Request #13676 · bitcoin/bitcoin · GitHubAsset 1Asset 1
< gribble>
https://github.com/bitcoin/bitcoin/issues/13662 | Explain when reindex-chainstate can be used instead of reindex by Sjors · Pull Request #13662 · bitcoin/bitcoin · GitHubAsset 1Asset 1
< sipa>
do we want to make the bot join in order to message here?
< sipa>
join/leave spam would be somewhat annoying, but not as bad as spam
< achow101>
most clients can hide join and leave messages, so I think that's fine
< midnightmagic>
Does the bot have a freenode account?
< midnightmagic>
If so, then +q $~a allows people to still join and watch, and worst case we get join/parts from the spammer bots.
< booyah>
sipa: bot must join/part becasue that is how github works?
< booyah>
possible solution: create #botx channel, have bot join say and part there. Setup a message relaying bot (tiny python script) to relay msgs from there to here, and the relay bot will be always joined
< sipa>
booyah: we added +n to this channel (which requires joining in order to speak) to combat spam
< midnightmagic>
the second bot would be present here as well and just speak it, I think is what he means.
< sipa>
yeah, i understand the suggestion - i don't have much of an opinion on it :)
< sipa>
i was just explaining it's not because how github works but because we have +n on
< midnightmagic>
ah
< sipa>
oh i see; i guess booyah understood that, but by "because that is how github works" booyah means as opposed to have it be continuously present
< sipa>
right
< booyah>
(yeah just afair github bot anyway was always joining and parting)