< wumpus>
jeremyrubin: well cloudabi executables are marked with a special "OS/ABI" type, so it's easy to identify them as such. Also it's recommended to use cloudabi-run to run them, which should not launch native executables
< gmaxwell>
wumpus: nice blog post btw.
< wumpus>
but yes it'd make sense to add a check to whatever processes submitted executables that it's really the right kind of executable
< wumpus>
gmaxwell: thanks :)
< gmaxwell>
wumpus: how is the kernel security for it implemented? assuming you're using its special libc and whatnot, does it just use seccomp mode 1 on Linux?
< gmaxwell>
ah, it's the Capsicum stuff.
< wumpus>
gmaxwell: the preferred approach is to have it run on top of capsicum, this is what is used on freebsd at least. There's also an emulator, which may use seccomp, not sure.
< gmaxwell>
it's a little sad that dns requests have to be external, since libc vulnerablities in dns handling is one of the things that bothers me more about our common attack surface. E.g. DNSseeds start returning a glibc dns 0day.
< wumpus>
well it could be implemented differently
< wumpus>
you could have something that passes in a ready-to-go UDP file descriptor for communicating to your preferred DNS server
< gmaxwell>
oh. hm yea, it's true.
< wumpus>
then use an internal DNS implementation
< gmaxwell>
or even a mini dns resolver process that just uses a non-sandboxed udp shim.
< wumpus>
(e.g. libevent's)
< wumpus>
exactly
< wumpus>
my initial usecase will be to have all outgoing network traffic go through tor, so it won't be doing any DNS lookups locally at all, only at the exit nodes I guess
< gmaxwell>
wumpus: for mlock, would it be possible for an external process to create an anonymous mmap, mlock it, then pass the descriptor over a unix domain socket to the sandboxed process?
< wumpus>
gmaxwell: yes, that's possible
< jonasschnelli>
What I once started is to use sipas crawler (bitcoin-seeder) and fill everything into a db, then use that db from djbdns (which is still one of the best dnsd IMO).
< wumpus>
gmaxwell: it's a bit scary though as the external process will be able to keep a reference, and have all your key data :)
< gmaxwell>
yes, though it could presumably also ptrace your process. and hopefully it wouldn't keep a reference. :)
< wumpus>
ideally we'd not want to store any key data in the process at all; e.g. interface to yet another module for that, which could be hw
< gmaxwell>
yea, indeed, should just be a signer process that has an interface like a hardware wallet. Sometimes it's an actual hardware wallet, other times it's a sandboxed process, SGX module, or whatnot.
< wumpus>
right
< wumpus>
jonasschnelli: to make a DNS server that can only look up bitcoind hosts?
< wumpus>
jonasschnelli: oh I get it, of course, you want to run an alternative DNS server as DNS seeder
< jonasschnelli>
wumpus: yes.
< jonasschnelli>
The crawler is great,... but not the dnsd part
< gmaxwell>
well we do use dns for things other than introduction.
< wumpus>
do we?
< gmaxwell>
sure, things like addnode can use it.
< gmaxwell>
and it's not a bad idea there simply because ... IPs change.
< jonasschnelli>
isn't that pure IPv4/v6?
< jonasschnelli>
we could or we can?
< nemgun>
Hello guys
< wumpus>
it's possible to pass some of the connect-to command line options (proxy, tor, etc) as hostnames.... but meh, there's no use for that in cloudabi because it won't be able to connect to those anyway. The right wya to do that would be to pass in a fd to a pipe or such...
< wumpus>
as for RPC you can do the lookups at the client side
< gmaxwell>
wumpus: ah, well I thought it would eventually be good to be able to support the complete feature set (except for some antiquated features like the notify interfaces which might be better done another way).
< wumpus>
well my goal with this experiment would be to make something better, sometimes that means making decisions to drop some functionality or do it some other way
< wumpus>
it'd be possible to emulate the complete old feature set on top of the sandbox, but that's not very interesting
< wumpus>
anyhow if addnode stores a name and does periodic DNS lookups, to track IP changes over time, there's something to be said that bitcoind needs to be able to initiate DNS lookups
< gmaxwell>
it does (well it will relookup again on a reconnect, e.g. after a disconnect under the old name...)
< wumpus>
simplest would be to use the "RESOLVE" [F0] SOCKS5 extension for that. This means the actual lookup can be done from another process (which could be sandboxed too)
< wumpus>
this also means that if the DNS resolver process is compromised, bitcoind itself isn't
< wumpus>
there's a bit of a bootstrapping problem here, ideally there would be a command 'instantiate a graph of cloudabi processes and connect them such and such'
< wumpus>
btw verify-commits.sh is broken again
< wumpus>
looks like it croaks on MarcoFalke's key
< bitcoin-git>
bitcoin/master eca550f Luke Dashjr: RPC/Wallet: Pass CWallet as pointer to helper functions
< bitcoin-git>
bitcoin/master d77ad6d Luke Dashjr: RPC: Do all wallet access through new GetWalletForJSONRPCRequest
< bitcoin-git>
bitcoin/master 2e518e3 Luke Dashjr: Move nWalletUnlockTime to CWallet::nRelockTime, and name timed task unique per CWallet
< bitcoin-git>
[bitcoin] laanwj closed pull request #8775: RPC refactoring: Access wallet using new GetWalletForJSONRPCRequest (master...multiwallet_prefactor_rpc) https://github.com/bitcoin/bitcoin/pull/8775
< paveljanik>
wumpus, dumping is fast, but loading is very slow and CPU consuming. And it consumes memory if there was a large mempool.dat. But I do not care enough ;-)
< wumpus>
I think there should be an option to bypass mempool loading/writing. And possibly it could parameter-interact with disable-blocks mode, as long as it's clearly documented.
< wumpus>
or maybe an option to disable the mempool completely? I don't know.
< wumpus>
I don't care enough either. It seems optimizing for something that is an incedigly rare edge case
< wumpus>
if you want to optimize actual bottlenecks, please pick my sha256 or crc32 optimization experiment :-)
< paveljanik>
Well, I'll first optimise my blocksonly gateway to the world ;-)
< wumpus>
both optimizations that I mention are very relevant to validating blocks, too
< paveljanik>
how do you satisfy multiarchitecture/multiOS with yasm?
< wumpus>
dunno?
< wumpus>
is yasm platform specific?
< wumpus>
as for architectures, use a general interface, and compile on the assembly appropriate for the architecture. For specific instruction sets it makes sense to do run-time detection.
< wumpus>
on x86 you can use the CPUID instruction for that (see for example the SDL2 implementation: https://github.com/spurious/SDL-mirror/blob/master/src/cpuinfo/SDL_cpuinfo.c), on ARM it's somewhat more difficult as the instruction to detect what instruction sets are available is privileged. So applications resort to OS-specific parsing of /dev/cpuinfo, or just trying out the instructions and catching SIGILL
< wumpus>
openssl, most notably, uses the latter approach, which is why you'lll always have to skip a SIGILL when running bitcoind in gdb on ARM
< bitcoin-git>
[bitcoin] sdaftuar opened pull request #9912: Optimize GetWitnessHash() for non-segwit transactions (master...2017-03-optimize-witness-hash) https://github.com/bitcoin/bitcoin/pull/9912
< gmaxwell>
What are the barriers on getting SSE2 SHA2 in use? The speedup for initial sync should be far from trivial now.
< gmaxwell>
(plus lowering latency of everything else we do)
< sipa>
what is the speedup factor?
< luke-jr>
why not SSE4 or AVX?
< gmaxwell>
If I'm not misremembering SSE2 was 95% of the speedup of SSE4 and AVX was not faster on the hardware we tested (and in fact AVX was slower on AMD). But just replace SSE2 in my comment with whatever made sense.
< gmaxwell>
sipa: before jonasschnelli benchmarked IBD with the change and it was 5% faster. IIRC... but now with assume valid a MUCH larger percentage of the time is in hashing.
< gmaxwell>
So "more than 5%", though I don't know how much more. It will only grow after your UTXO changes.