< warren>
Regarding the mailing list migration. We are finding groups.io to be unacceptable as a backend solution. Meanwhile it seems the mailman2 upstream project actually isn't dead. The problem in the past however has been that no third party wants to host these dev mailing lists because it attracted heavy DoS and other types of abuse attacks. So we may be forced into self-hosting our own mailman2. That includes paying for a server with DDoS
< warren>
protection and also paying a sysadmin to keep adjusting the configurations to stop different attack strategies that people keep doing.
< warren>
The lack of a central org to pay for this is a problem. But I suppose <orgs> and <people> can donate to whoever maintains it.
< warren>
The alternative to that would be to just switch to Google Groups and never again worry about maintenance. But people hated that option when this was last discussed back in 2015.
< warren>
The DoS attacks with shifting strategies with the goal of getting this list server on blacklists is the really hard part to deal with.
< warren>
The plan since last year had been to move to the known non-optimal groups.io as list backend, but to host our own mailing list archives at permalinks outside of any service provider. That way the backend could change and at least history isn't lost. Permalinks are important to keep a record of the past intact, for patent priority, etc.
< warren>
Since then we had some major delays (partially me, mostly waiting for other people). Now upon a closer look at groups.io I think it is really bad.
< warren>
It is an ongoing non-trivial effort to maintain a list backend and to prevent abuse. It is also difficult to keep that server's IP address from not getting blacklisted.
< warren>
Nobody here even wants to discuss or help. So now I'm thinking move the list backend to Google Groups and be done with this. I really don't want to deal with this either.
< warren>
So I propose moving the dev lists to Google Groups. We'll also maintain archives on a domain outside of Google. Anyone can replicate that archive. Permalinks should be there. We're asking for the old domain name to auto-redirect to there too.
< warren>
Last time this was a debate in 2014/2015 people really hated the Google Groups option. But nobody is willing to put any effort into this. Soon LF will kick us off their server. They asked us to move 2 years ago now.
< warren>
Express an opinion here in public. Or the list will just disappear unexpectedly and we'll be rushed into a shitty solution.
< warren>
An experienced sysadmin is offering to maintain a self-hosted solution and to deal with the DoS attacks for pay if people really want as little change as possible.
< warren>
Somebody please express any opinion?
< sipa>
warren: i'm surprised tha5 self hosting is such an issue. what do other organizations do, like apache?
< jonasschnelli>
pushed the 0.19.0rc1 osx detached signature (waiting for cfields win now)
< sipa>
warren: i'm sure resources can be found to handle the logistics if there is a proposed solutions
< sipa>
but perhaps the ML itself is a better place to look for this?
< sipa>
do you know what lightning-dev will do, btw?
< sipa>
they must have the same problem
< warren>
sipa: same problem, they would just follow. nobody wanted to discuss.
< warren>
sipa: maintaing a MTA, preventing abuse in the face of changing attack strategies and avoiding blacklisting is on-going effort. Also the Open Source automatic filtering solutions all suck. Nearly all the authors of things like spamassassin were hired away and stopped working on it 10+ years ago.
< sipa>
warren: have you tried asking optech for help? i think many people apart from a number of developers here even know there is an issue
< sipa>
*don't think
< warren>
sipa: sure can ask, but only sysadmins who tried to maintain this know how much of a pain this is.
< sipa>
warren: well i doubt you'll find many sysadmins by asking developers :)
< sipa>
ping moneyball, jnewbery, harding ^
< warren>
sipa: I was the only person keeping the spamassassin project alive for a few years, I maintained a MTA to dogfood both during and after working at Red Hat. 2010-2012 I wrote this blog for sysadmins to configure and tune spamassassin properly. http://spamtips.org had about 10k unique readers per year. I gave up in frustration when people only complained and zero people donated in two years. I then stopped maintaining my own MTA.
< warren>
sipa: there's an experienced sysadmin in this channel willing to put up with this pain for pay.
< warren>
A particular technical issue with groups.io that is very frustrating is it munges the message-id
< midnightmagic>
gah wrecking reply threads.
< warren>
contrary to what I'd been told, mailman2 is in fact not dead upstream, but it requires additional efforts to prevent abuse, re-patch with Google CAPTCHA after every release, on top of all the annoying effort it takes to maintain a MTA that doesn't get blackholed
< jrayhawk>
groups.io rewrites all Message-Ids: to new @groups.io Message-Ids:. They optionally also provide an option to exclusively send you your own original Message-Ids, presumably to avoid duplicating messages sent to yourself.
< warren>
for a moment I thought about going back to maintaining a MTA again, but I don't need this kind of stress in my life.
< jrayhawk>
Other CC'd individuals who aren't interested in duplicate messages or being able to build coherent threads can apparently go screw themselves (???)
< jrayhawk>
Additionally groups.io provides no canonical posting date outside of a very fragile X-Received header, which degrades quality of external indexing.
< warren>
Other options for self-hosting: mailman3 doesn't scale, people hate it, other people reportedly hate its archiver hyperkitty
< warren>
So unless somebody WHO KNOWS WHAT THEY ARE DOING has a better option, I suggest we just move it to Google Groups and forget about maintenance headaches forever.
< warren>
Or if people don't want that then this community needs to pay the sysadmin to maintain a self-hosted mailman2. Then very little about it changes other than a different address. Actually it would be better than now because LF hasn't updated their mailman or tried to fix it for 2-3 years now.
< warren>
It will still be a headache to maintain
< sipa>
warren: send a mail to the list about it
< warren>
I'm exploring one last attempt of a 3rd party host of existing FOSS mailing lists to take care of it.
< warren>
It would be helpful if <proposed non-profit who hosts existing FOSS dev lists> could receive donations to take on the extra burden. Folks are contacting them to ask about feasibility.
< bitcoin-git>
[bitcoin] hebasto opened pull request #17059: util: Simplify path argument for CBlockTreeDB ctor (master...20191005-blocks-index) https://github.com/bitcoin/bitcoin/pull/17059
< bitcoin-git>
[bitcoin] martinus opened pull request #17060: Cache 26% more coins: Reduce CCoinsMap::value_type from 96 to 76 bytes (master...2019-09-more-compact-Coin) https://github.com/bitcoin/bitcoin/pull/17060
< captjakk>
does anyone have an idea of what the write throughput (persistence wise is) is for bitcoind
< captjakk>
motivation: is the write bandwidth on a microsd sufficient to keep up with the network
< gwillen>
well, the current block size average is around 1MB, and you get one around every 10 minutes, which gives a block write bandwidth of 1.7 kilobytes per second
< gwillen>
the absolute slowest SD cards are Class 2, which is 2MB/s
< gwillen>
and most of the ones you get now are at least UHS-1, which is at least 10 MB/s
< gwillen>
so you've got a good margin on write bandwidth, and the thing you should probably worry about is wear
< gwillen>
the internet tells me 100,000 write cycles is the typical limit for flash -- because of wear levelling, a bigger card will last long here
< gwillen>
longer*
< captjakk>
interesting, I would have expected though since most of the blocks get written once and then basically ignored from that point forward
< captjakk>
which wouldn't cycle the writes terribly much for the archive data
< captjakk>
the main cycling of writes is for turnover on the UTXO set
< captjakk>
which only occupies about 3-4GB iirc
< gwillen>
oh, that's a good point, derp
< gwillen>
indeed, wear shouldn't be an issue at all, _except_ if you're running a whole system off an sd card, your system logs will create way more wear than the blocks will (from personal experience)
< elichai2>
I implemented siphash using SSE4.1 but it's in practice way slower for `SipHashUint256Extra`. I'm curious if there's a place to document failed optimizations so people can know what was tried and failed lol
< bitcoin-git>
[bitcoin] martinus opened pull request #17063: [RFC] Tests: use boost test framework for benchmarks (master...2019-10-refactor-benchmark-framework) https://github.com/bitcoin/bitcoin/pull/17063
< emilengler>
Btw, why are there no bitcoin core project cloaks on freenode?