< meshcollider>
sipa: re #11693 do you know if there is any way to currently signrawtransaction for P2SH-P2WSH ? Or should I open a PR to add a witness program key to that call
< meshcollider>
Or is there something else planned
< sipa>
meshcollider: hmm, that is surprising
< sipa>
meshcollider: maybe we should just automatically add the p2sh version of each script passed to signrawtransaction
< sipa>
that should work without api break
< meshcollider>
Mmm so you would just pass the internal WSH script in the scriptSig?
< meshcollider>
JSON*
< sipa>
right
< sipa>
hmm, but it would need to guess the witness version
< meshcollider>
sipa: it's already a JASON object so adding an extra key wouldn't break it anyway would it
< meshcollider>
Oops did that resend
< sipa>
okay
< sipa>
good
< gmaxwell>
from a usability perspective, having to have some extra key isn't super friendly, if it can be reasonably avoided.
< gmaxwell>
(these interfaces do exist for thigns other than driving test harnesses, you know)
< meshcollider>
I don't know how it could be avoided without instead passing in the witness version, like sipa mentioned
< sipa>
or alternatively, doing it automatically for all known witness types and versions
< meshcollider>
Why does the signrawtransaction help text already say "redeemScript": "hex", (string, required for P2SH or P2WSH) redeem script
< meshcollider>
Oh don't worry, that's P2WSH not P2SH-P2WSH
< gmaxwell>
whatever we do, it should be the case the you can form a valid signraw line using nothing other than a simple reformatting of listunspent's output.
< meshcollider>
What is the redeemScript output from listunspent for P2SH-P2WSH
< meshcollider>
I assume it doesn't currently output the witness script at all, which will need to be modified too
< sipa>
gmaxwell: that's not really possible, and already not true for P2SH
< sipa>
at least not for watch-only outputs
< gmaxwell>
sipa: if you've imported the script it should be possible (I have no watch only p2sh so I don't know if it does)
< sipa>
gmaxwell: it won't work
< gmaxwell>
We should fix that then.
< sipa>
i mean it won't work when passing in explicit private keys
< sipa>
because then it uses the keystore consteucted from the rpc arguments rather than your wallet's keystore
< sipa>
part of this confusion is solved by finally splitting up signrawtransaction into a wallet version and a utility version (achow has a pr i think)
< gmaxwell>
the flaw here is that our private key encodings don't represent (or imply) the redeemscript.
< sipa>
in the wallet version, things already work fine and nothing is needed for p2sh-p2wsk
< sipa>
in the non-wallet version you inherently need to pass in the solving information somehow
< sipa>
indeed one way is to encode it in the private key
< gmaxwell>
or just declare that it's already encoded in the private key, and expand each private key all known ways; but that doesn't work e.g. for multisig.
< sipa>
but that's not generally possible for all output types (and specifically won't work for.any multisig kinda thing, which is exactly where p2wsh is used), so other ways must exist as well
< sipa>
right
< gmaxwell>
e.g. current private keys = {p2pk,p2pkh,p2sh-one-key,p2wpkhv0,p2wpkhv0-p2sh}
< gmaxwell>
we have the redeemscript for each input we're going to sign as an argument, so it could meet in the middle with the private keys. ugh.
< sipa>
yeah
< sipa>
whatwver encoding is used for the solving information, listunspent should probably be made to report it
< sipa>
if known
< Chris_Stewart_5>
gmaxwell: So basically this would look like attaching an extra byte of data to the current format to indicate the script it corresponds to>
< Chris_Stewart_5>
?
< Chris_Stewart_5>
and some sort of standardized scheme for standard script types I guess
< meshcollider>
So if I just made listunspent just report witness script for P2SH-P2WSH outputs and then signrawtransaction accept it that would be simple right
< meshcollider>
That's the cleanest way i can see to ensure P2SH-P2WSH multisig works for example, which is what brought on this discussion
< sipa>
while you're at it, also add redeemscript for P2SH?
< meshcollider>
To listunspent?
< sipa>
yes, it's also needed in signrawtransaction when giving private keys manually
< meshcollider>
sipa: I thought listunspent already had it
< eck>
random question: if leveldb already has its own WAL, then why does bitcoin need an in-memory cache/buffer for utxo updates
< wumpus>
eck: because we can do it more efficient because we know the properties of the data and access patterns
< wumpus>
eck: leveldb's caching is virtually useless for our use case, so we minimize their caches
< wumpus>
why does a database implement its own caching if the OS already has a page cache'
< wumpus>
would be similar :)
< eck>
i don't care/know that much about how leveldb implements read caches, but for writes, it seems really problematic you have multiple caching layrs
< eck>
in this example you have bitcoin cache + leveldb cache + kernel page cache
< eck>
that's a lot of caching layers
< wumpus>
why would that be problematic? it's pretty much how modern platforms work, your CPU also has caches as different levels
< sipa>
eck: there are a number of reasons why our caching provides functionality on top of LevelDB's
< sipa>
one is that LevelDB inherently deals with serialized records, while our caching layer stores information using our native in-memory formats
< sipa>
serialization and deserialization have a nontrivial cost
< wumpus>
in any case, if you find a way to improve performance, go for it
< sipa>
furthermore, we're able to exploit a very significant property of our data set: entries are written once, and deleted once - and almost never overwritten in between
< eck>
maybe I can or cannot, just trying to make sure I understand the current state of affaris:
< eck>
as i understand it, by default a full node today will flush to disk as infrequently as once every 24h
< sipa>
we keep track in our cache whether an entry exists in the caching layer below, and if not (= it's a freshly created entry) which gets deleted (utxo gets spent), we simply delete it from memory, without it ever needing to touch disk
< sipa>
this optimization saves us 90% of all I/O or more, depending on cache sizes
< sipa>
because many UTXO entries get spent quickly after being created
< bitcoin-git>
bitcoin/master 5b56ec9 Wladimir J. van der Laan: qt: refactor: Use absolute include paths in .ui files
< bitcoin-git>
bitcoin/master 0c71521 Wladimir J. van der Laan: build: Remove -I for everything but project root...
< eck>
could you point me (rougly) to the part of the code i woudl oook at?
< wumpus>
yes, leveldb does not do this for batches
< sipa>
eck: coins.cpp
< wumpus>
so if you queue both adds, updates, and deletes, they will actually get executed
< bitcoin-git>
[bitcoin] laanwj closed pull request #11651: refactor: Make all #includes relative to project root (laanwj, MeshCollider, ryanofsky) (master...201711_absolute_includes) https://github.com/bitcoin/bitcoin/pull/11651
< sipa>
yes, LevelDB's "batch" structure is literally an std::string
< wumpus>
also leveldb's batch storage format is really inefficient with regard to memroy allocation, large consecutive memory areas
< wumpus>
right
< sipa>
with serialized write/erase operations in it
< gmaxwell>
eck: the infrequency is basically unrelated to the structure, it could continiously flush, now that we have non-atomic flushing... that just hasn't been implemented yet.
< gmaxwell>
and in general we want to delay flushing as much as possible in order to suppress writes from ever happening in the first place.
< eck>
thanks! in the past I wrote my own (simpler obviously) C++ implementation of an ss-table-like structure, and I'm trying to confirm that the whole caching i/o layer in bitcoind works as I expect it would
< sipa>
right, the idea is to move the flushing to a background process that continuously flushes the oldest dirty cache entries
< sipa>
but never gets too close to 'now', as to not interfere with our freshness optimization
< sipa>
eck: cool
< eck>
thanks everyone for your help, I will report back if I have further conclusions or questions
< sipa>
yw!
< eck>
related, if I wanted to follow up with someone speifically about the storage layer, who is an expert?
< gmaxwell>
just ask here.
< gmaxwell>
other people would learn from your questions.
< eck>
alright, at a high level i want to know why bitcoin has a caching layer at all: there is already some caching in the kernel page cache, leveldb has some caching og it its own, and then bitcoin itself will cache data for up to 24h
< wumpus>
that's what sipa just explained
< eck>
from my superfcial understanding, i would just write directly to leveldb and let it take care of the rest
< wumpus>
that was the first thing that was tried, and perf was abysmal
< gmaxwell>
eck: you can do that, it takes about a week to sync the chain that way on typical hardware.
< wumpus>
that might work better with other databases, but not leveldb
< gmaxwell>
everything else we've tried was basically an order of magnitude slower than leveldb.
< wumpus>
the best (only) way to get feeling for it is to experiment, it's really hard to beat the performance of the current solution
< eck>
interesting, I would like to / am willin g to repeat the experiment to verify it locally
< gmaxwell>
eck: you can just set the dbcache to a minimal value and see the result for 95% of the effect.
< gmaxwell>
eck: syncing the chain with every operation going to the database requires about 1 billion database updates.
< wumpus>
maybe that's useful at least for seeing what files to touch...
< eck>
generally though,i would account for the wallet db taking basically 0 time, right?
< wumpus>
LMDB seemed promising but I never got spectacular results, maybe I was just using it wrong, and maybe the approach of not caching at the bitcoin level would work better with it
< gmaxwell>
eck: I can't figure out why you think the wallet database would be involved at all.
< wumpus>
wallet has nothing to do with this, when you benchmark this, I suggest you disable it compile-time
< gmaxwell>
The wallet database uses reasources when you have wallets with loads of transactions and keys, otherwise it does nothing.
< eck>
i don't care about the wallet database, it's slow but it's smalll
< wumpus>
it's also only read at startup, and written when the wallet actually changes, which is infrequent for most people
< eck>
my interest recently is syncing a g1-small GCE instance
< gmaxwell>
in any case, to sync in three hours (which we do, on a fast desktop at least with dbcache cranked) without a dbcache would require sustaining 100k operations per second, which is not realistic except on specialized hardware (e.g. nvme raid or whatever).
< eck>
it has 1.5GB of memory, and essentially 0 disk I/O
< wumpus>
it has terrible I/O, disabling caching is certainly not what you should look at
< eck>
my instance on core right now is synicin at < 5% a *day*, which seems like there is some fundamental horkage in some layer
< wumpus>
sync on a faster machine then copy over the state
< eck>
yeah maybe
< gmaxwell>
I think you might just be underestimating how much work is involved in syncing and how slow those instances are. :)
< wumpus>
w/ the cloud thing, you could just hire one of the large instances for a day or so
< gmaxwell>
with 1.5GB memory you may be able to increase the cache further without crashing, it will speed it up.
< eck>
from what I can tell, on GCE they give you I/O capacity base on the disk size
< wumpus>
this is comparable to running a node on e.g. rpi, they can keep up, but doing initial sync on them takes extreme amounts of patience
< eck>
so if you want 200 GB, you're basically on teh bottom wrung, even if you have a ton of cpu/memory
< gmaxwell>
But there is just an utterly stupendous amount of work required, ... the software is highly optimized (and sure, there are also still many things that could be done to make it faster). .. but e.g. compared to ethereum the amount of blockchain bitcoin core processes per second is something like two orders of magnitude greater. ( https://anduck.net/bitcoincore_vs_geth_full_node_stats.png )
< wumpus>
if it's a vm maybe you can temporary increase the amount if memory, then sync with dbcache of 4000, then i/o will not be touched until it's done
< eck>
thanks all, I will definitely be in here asking about this gain
< gmaxwell>
indeed, the only IO with a huge db cache is just writing blocks out to disk and the final flush.
< eck>
I will ask more once I've done more research
< wumpus>
oh yes it will need to write out the blocks, but that's linear and granular, not seek-heavy database i/o I meant
< meshcollider>
To add witnessScript to listunspent output, how do we retrieve a CScript from the wallet using WitnessV0ScriptHash if the CScripts are indexed by CScriptID which is Hash160 not SHA?
< sipa>
aha!
< sipa>
Hash160(x) = RIPEMD160(SHA256(x))
< sipa>
so given y=SHA256(x) you can compute Hash160(x) as RIPEMD160(y)
< bitcoin-git>
bitcoin/master 069215e practicalswift: Initialize recently introduced non-static class member lastCycles to zero in constructor...
< bitcoin-git>
bitcoin/master 084f52f Wladimir J. van der Laan: Merge #11654: tests: Initialize recently introduced non-static class member lastCycles to zero in constructor...
< bitcoin-git>
[bitcoin] laanwj closed pull request #11654: tests: Initialize recently introduced non-static class member lastCycles to zero in constructor (master...uninitialized-members) https://github.com/bitcoin/bitcoin/pull/11654
< asu>
hi, all
< asu>
i has a question.
< asu>
i want to build a bitcoin p2p trade web
< asu>
buy i found if my customer send 0.01bitcoin to another, the fee is 0.00027 bitcoin
< asu>
but in localbitcoins.com, the fee is zero
< asu>
how can i do it? thanks
< asu>
who can help me
< kinlo>
asu: that question is for #bitcoin, not for here
< wumpus>
BlueMatt: I think it'd make sense, but I don't really want to turn it into a controversial topic
< BlueMatt>
oh I dont think anyone cares *that* much, I was just curious if you want more pr volume
< BlueMatt>
its also a github md file, not like we need to have big review cycles on it.......
< wumpus>
right, a change to a github md isn't so bad with regards to PR volume
< BlueMatt>
k, I'll tweak it again
< wumpus>
thanks :)
< bitcoin-git>
[bitcoin] TheBlueMatt opened pull request #11706: Make default issue text all comments to make issues more readable (master...2017-11-shorter-default-issue-redux) https://github.com/bitcoin/bitcoin/pull/11706
< gmaxwell>
will be back in 10 minutes, maybe the meeting won't be over by then. :P
< wumpus>
#topic high priority for review
< BlueMatt>
new high-priority for me: #11639
< gribble>
https://github.com/bitcoin/bitcoin/issues/11639 | Rewrite the interface between validation and net_processing wrt DoS by TheBlueMatt · Pull Request #11639 · bitcoin/bitcoin · GitHub
< BlueMatt>
also probably worth a post-merge review: #10286 (note that this will likely make lots of open wallet-rpc change conflict silently - you need to add the new BlockUntilSyncedToCurrentChain call in some wallet rpc functions as boiler plate, see dev docs for more)
< promag>
Rpc console still only for 1st wallet right?
< luke-jr>
promag: that PR has an independent combobox for the debug window
< luke-jr>
(including a "no wallet" option)
< promag>
Should rebase on dynamic wallet loading? Or vice-versa?
< wumpus>
#topic rpc console for multi wallet
< jonasschnelli>
The dropdown seems okay isch.
< luke-jr>
promag: IMO GUI should go before dynamic loading
< jonasschnelli>
Ideally we would have a higher-level visual selector
< promag>
Kk
< luke-jr>
jonasschnelli: ?
< jonasschnelli>
luke-jr: agree
< jonasschnelli>
luke-jr: it confusing to have a wallet level switch in the console
< jtimon>
what's wrong with the combobox ?
< jonasschnelli>
But I don't see another simple way
< promag>
One thing that bothers me with the combo is that the gui state is lost
< luke-jr>
maybe improvements there can be made after merge, if someone thinks of a better way
< luke-jr>
promag: ?
< achow101>
I think it might be confusing to users to have the debug window possibly be for a different wallet than the main wallet gui
< wumpus>
the combobox is ok
< jonasschnelli>
I think its an acceptable first step
< promag>
Like list scroll position, selection, focus, etc
< sipa>
pieter was here
< wumpus>
the debug window is supposed to be separate from the main GUI, having it influence what wallet is selected is even more confusing
< jtimon>
I think it's perfectly fine for the debug console to be flexible like this. seems just handy to put it there
< wumpus>
yes
< promag>
Another option is one tab per wallet
< wumpus>
no, please not
< luke-jr>
maybe (post-merge) an idea might be to have a red alert icon next to the combobox if it doesn't match the main window
< achow101>
I was thinking that when you first opened the debug window it could default to the wallet that was in use in the main window
< wumpus>
meh
< achow101>
then users can change the wallet if they want to
< jonasschnelli>
I think the dropbox is still the best solution on the table,... (even if not ideal)
< jtimon>
that sounds reasonable to me
< wumpus>
I really think having the debug window and main window interact in that way is a mess both in code and in interaction, but anyhow
< wumpus>
okay, any other topic?
< luke-jr>
sounds like we at least agree it's a post-merge topic XD
< jonasschnelli>
ack
< jtimon>
oh, if it's a mess in the code, I'm not sure it's worth it. I'll shut up
< promag>
wumpus: btw why not tabs?
< wumpus>
promag: multiple tabs with the same console just pointing at a different wallet sounds terrible to me
< jtimon>
spaces are better
< jonasschnelli>
promag: most calls are pure node calls...
< * jtimon>
hidfdes
< wumpus>
promag: the tabs are supposed to be for essentially different things
< promag>
At least you keep track of the correct log
< wumpus>
e.g. more charts, more pages of debug info, etc
< jnewbery>
promag: multiwallet comes first, dynamic loading later
< jnewbery>
*multiwallet GUI comes first
< promag>
Anyway, ack on the order
< luke-jr>
promag: perhaps a log entry after you execute a command on a different wallet than previously (post-merge stuff)
< promag>
Ok ok
< wumpus>
why not a command to switch between wallets, btw?
< wumpus>
the combobox is great to show what the current wallet is, but shouldn't the wallet be switchable with typing?
< luke-jr>
/wallet <name> ?
< wumpus>
for ex.
< jonasschnelli>
yes... ideally it would be stateless.
< jonasschnelli>
to ensure one is not executing on the wrong wallet
< achow101>
so it would be a gui only command?
< jonasschnelli>
wallet:xyz getnewaddress
< wumpus>
achow101: yes
< jonasschnelli>
if wallet:<filename> is missing, we get the standard rpcish reject
< wumpus>
jonasschnelli: type the wallet name for every command? yes, maybe
< promag>
It could be part of the "prompt"
< MarcoFalke>
Needs autocomplete!
< jonasschnelli>
I think the wallet-selected-state can be dangerous
< wumpus>
jonasschnelli: that's absolutely safest
< wumpus>
jonasschnelli: agree
< jonasschnelli>
and it's RPC like
< wumpus>
yes
< jonasschnelli>
one can still use arrow-up edit
< jtimon>
a gui only command doesn't feel right
< luke-jr>
nesting is already GUI-only
< wumpus>
jtimon: no, agree, jonasschnelli's proposal to make it stateless and have to provide it for every command is better, that's the same as needs tobe done for bitcoin-cli
< jonasschnelli>
yes. It's fine
< luke-jr>
is this still post-merge, or have we un-concept-ack'd the MW GUI PR?
< promag>
Even when there is 1 wallet only?
< MarcoFalke>
promag: No
< wumpus>
promag: that's the exception
< jonasschnelli>
luke-jr: both would be okay for me (post merge or now)
< wumpus>
if it is unambigious then why not
< wumpus>
wallet needs to be provided if multiple wallets are loaded
< promag>
Ack
< luke-jr>
wumpus: because it'd be really annoying to use?
< wumpus>
if no wallet is loaded, there's no problem, if one wallet is loaded, then it's clear which one is meant
< wumpus>
if mutliple are loaded then wallet commands are ambigious
< promag>
It's the same with cli
< wumpus>
yes, it's the same with bitcoin-cli
< jonasschnelli>
It's maybe annoying... but it's the wallet. Safety first
< luke-jr>
cli is just a testing tool though; it doesn't need to be convenient
< gmaxwell>
why wouldn't the debug window just have a combo box
< luke-jr>
gmaxwell: that's the current code
< jonasschnelli>
gmaxwell: I think you will quickly choose the wrong wallet
< wumpus>
gmaxwell: it's somewhat dangerous; easy to type a command with the wrong one selected
< jtimon>
gmaxwell: some people are worried about a state, not sure what the problem is either
< gmaxwell>
luke-jr: that is not true. cli is probably about as frequently used for using the software as the gui (this probably says some unfortunate things about the gui, but.. :P )
< luke-jr>
could do both, I guess
< luke-jr>
gmaxwell: I highly doubt that!
< achow101>
luke-jr: I think there could be some weird interactions with doing both
< wumpus>
gmaxwell: there is no clear visual link between what you type and the combobox, though it could be somehow improved by logging in big colorful letters when a different wallet is selected
< luke-jr>
both = combobox with in-command override only when no-wallet selected
< wumpus>
e.g. ============ current wallet: blabla.dat ===============
< gmaxwell>
wumpus: thats a point, the prompt to could also show the wallet.
< wumpus>
gmaxwell: yes, indeed
< gmaxwell>
and there could be a line written in when it chages, like that.
< jtimon>
how is it going to be with the cli again?
< luke-jr>
jtimon: no changes needed there
< wumpus>
jtimon: for the cli you have to provide the wallet name on every call to select the endpoint ,if it's ambigious, nothing will change there
< gmaxwell>
jtimon: cli makes you specify it as a dashed argument to bitcoin-cli, which is a bit obnoxious but works.
< wumpus>
decision is to be made about the console
< wumpus>
but seems a combobox will do for now
< wumpus>
so leave it like that for now luke-jr
< luke-jr>
k
< promag>
Next?
< jtimon>
I see, thanks. just like you have to provide testnet or regtest every time but you don't need that in the GUI
< wumpus>
jtimon: yep
< wumpus>
GUI can keep state for you that the cli cannot
< wumpus>
because it 'captures' the user, unlike a command that's launched every time
< wumpus>
yes, other topics?
< achow101>
topic suggestion: encrypted wallets by default
< morcos>
can someone open an issue about deciding wallet access from the console, i think shipping with it as i understand it to be now seems terrible, but i agree no reason to hold up progress on merging
< jonasschnelli>
achow101: with an option to unencrypt later?
< achow101>
jonasschnelli: I guess?
< sipa>
wumpus: why what?
< wumpus>
why encrypt the wallet by default?
< jonasschnelli>
achow101: I think that would be great.
< gmaxwell>
If you have users encrypt wallets when they open one without any value in it they will reliably lose the key. The positive confirmation that the user is backed up like electrum has reduces that sort of risk.
< wumpus>
it forces people to choose a passphrase which they'll probably forget
< achow101>
a lot of wallet software do this now and I don't think people necessarily realize that their wallets are unencrypted until they go to the encrypt wallet option or rpc
< wumpus>
I think most people lose money because of losing wallets or losing passphrases not theft
< wumpus>
what thread model does encrypting wallets protect against anyhow?
< jonasschnelli>
that true on the other hand
< jonasschnelli>
Those who have access to support ticket systems of consumer wallets do know that
< luke-jr>
wumpus: bad PR
< gmaxwell>
Wallet encryption is mostly a tool for people to lose their money but feel better about it because its their own fault. The great advantage of wallet encryption by default, as I'd see it, is resolving this mess of having to preserve unencrypted keys.
< morcos>
couldn't we encrypt the wallet by default but not create the wallet by default
< morcos>
so you solve the problem of them just clicking through the encryption aspect
< achow101>
morcos: that was the idea I was thinking about
< gmaxwell>
But for that advantage I would recommend a late initilization that doesn't create a wallet until you ask for an address... or go to encrypt it.
< achow101>
you don't make the wallet until it is actually used, and only then do you prompt the user to make a wallet
< wumpus>
I mean, the only use for encrypting wallets I see is: other people use your computer, and you're afraid of them copying the wallet but not installing a keylogger
< gmaxwell>
+1 on the late initilization.
< wumpus>
I don't think it protects against any other attacks
< morcos>
wumpus: you dont think its useful for backups?
< gmaxwell>
wumpus: well I really like encryption so that I know that I'm not accidentally going to send funds, but for that it's sufficient to make the key "yes" :P
< luke-jr>
morcos: for backups you really want to encrypt the whole thing anyway
< morcos>
i suppose, maybe backups wasn't the right word
< achow101>
it doesn't work right now
< morcos>
maybe i meant having the wallet to check on things but not worrying too much about it
< wumpus>
or maybe the case where e.g. malware in the browser sandbox can grab a fixed file from your computer, but there's no persistent access
< achow101>
also encryption reduces the file size by like half because unencrypted keys are massive for some reason
< wumpus>
another thing that will cause confusion is that for other wallets, the passphrase is the seed
< luke-jr>
wumpus: even when it was introduced, it was acknowledged as mostly just a PR stunt
< wumpus>
so people will think that only keeping the passphrase is enough to keep access to their funds
< jtimon>
gmaxwell: I was actually scared to suggest a default key for "resolving this mess of having to preserve unencrypted keys"
< wumpus>
there are already peple making that mistake now but it's rarer
< wumpus>
(because you only have to choose it explicitly)
< luke-jr>
achow101: huh? how?
< gmaxwell>
+1 for late init, +1 for positive confirmation recovery backup (like electrum); -1 for more pressure to encrypt unless the last step is done, +1 for it if the last step is done.
< morcos>
also, this might sound stupid, but if you have a Core-encrypted wallet, you at least know the balance, so you know whether it's worth trying to figure out how to unencrypt it
< wumpus>
so no, I think focing people to choose a passphrase when first creating their wallet is a bad idea
< achow101>
luke-jr: encrypted keys are way smaller than unencrypted ones
< morcos>
+1 gmaxwells +/-1's
< luke-jr>
how is that even possible?
< promag>
Sorry have to be afk
< gmaxwell>
luke-jr: because the unencryted keys use some brain damaged openssl encoding
< gmaxwell>
that encludes all the curve parameters.
< wumpus>
that's just an implementation detail htough; unencrypted keys could be stored smaller, too
< achow101>
luke-jr: the format. unencrypted keys are DER format or something. they have the curve params in them
< wumpus>
we could encrypt the wallet by default, with an empty passphrase
< luke-jr>
ew
< gmaxwell>
right, thats a reason to change the format, not a reason to encrypt.
< sipa>
achow101: they have field params, curve params, generator, public key and private key in them :)
< sipa>
and all of that in inefficient DER
< sipa>
279 bytes total, iirc
< wumpus>
yes it's terrible
< wumpus>
and doesn't help with anything, if you're going to store the keys in redundant format at least pad it with something that provides error correction
< BlueMatt>
I mean its error correction in case we forget our curve parameters...or something
< luke-jr>
XD
< sipa>
BlueMatt: we actually hardly look at it
< gmaxwell>
wumpus: What are your thoughts on, long term: delayed creation, at create time in the GUI force the user to write down a recovery code (like electrum does; force via reentry and copy/paste jamming).. and have a checkbox to encrypt there too? recovery code would greatly offset all risks of loss, including lost the passphrase.
< luke-jr>
at that size, just store 8 copies of it
< wumpus>
gmaxwell: the recovery code would be the HD seed?
< gmaxwell>
luke-jr: storing N copies of a key right next to each other hardly helps since disks tend to die a physical sector at a time.
< achow101>
gmaxwell: recovery code as in something like bip39?
< gmaxwell>
wumpus: yea, an encoding of the HD seed.
< wumpus>
gmaxwell: that sounds great to me
< morcos>
gmaxwell: encryption using recovery code?
< luke-jr>
gmaxwell: sure, but in that case you're screwed with checksums too
< gmaxwell>
achow101: not bip39 as it's a brainwallet scheme that can't encode arbritary data, but yes.
< morcos>
i also like that idea, but i worry about the importing of private keys... we'd have to put in a whole lot of warnings about that
< wumpus>
achow101: more like other wallets lke electrum's seed phrase
< achow101>
wumpus: yes, I would prefer using Electrum's scheme
< wumpus>
achow101: (there's a BIP for it but I don't know the number)
< gmaxwell>
morcos: I think we need to get to having an import tainted flag on wallets, and warnings about that.
< achow101>
that's what we plan to do for Armory
< luke-jr>
morcos: importing private keys is already considered dangerous and "never do this"
< wumpus>
gmaxwell: I also greatly like the idea of not creating a wallet by default, so starting in no-wallet mode
< jtimon>
so what's wrong with the "yes"/default/empty passphrase/key?
< jonasschnelli>
the recovery phrase would be unencrypted?
< gmaxwell>
achow101: ugg electrum itself. can't encode arbritary data, so it can't work with existing wallets. at least it's better than bip39.
< achow101>
jonasschnelli: it would have to be to be able to recover from forgotten passwords
< achow101>
gmaxwell: it can't? (I haven't really looked at it)
< jtimon>
jonasschnelli: yes, would be public knowledge (and for the user it would be like if none was set) unless you actively set one
< jonasschnelli>
achow101: I just worry about people storing those recovery phrases on phones and "plaintext "papers
< gmaxwell>
jonasschnelli: I have mixed feelings about that. I think a best practice is to have your recovery keys encrypted with a WEAK key, like that insecure password your whole family knows; and there is no risk of it being forgotten... but which a burgler would likely be thwarted, but thats too complex to communicate.
< gmaxwell>
jonasschnelli: but we should realize that risk of users losing a strong password is likely orders of magnitude more likely than a local in person attack.
< wumpus>
it gets quite complex to manage if the recovery key is encrypted too
< * BlueMatt>
notes that if we do some kind of default-encryption-with-weak-password we should have a clear tag on keys to help out the "shitsihitshit, please scan entire raw disk for anything that looks like a key" use-case
< jonasschnelli>
gmaxwell: Indeed. Though people who can take care of a passphase should not be punished
< wumpus>
there's the recovery key passphrase, the wallet passphrase,...
< gmaxwell>
achow101: unless I'm confused (always likely) it's just a minor fixup of BIP39.
< luke-jr>
BlueMatt: +1
< wumpus>
BlueMatt: that's where the redundant key format is useful :)
< wumpus>
BlueMatt: it greatly helps efficiently scanning for private keys on a disk :p
< BlueMatt>
heh, I know
< gmaxwell>
BlueMatt: yea, sure, anything key format should have e.g. somethin like the network magic then the private key then a 64 bit crc... and then its cheap to scan the media looking for it.
< wumpus>
I don't think you can do a similar thing for the encrypted keys right now
< wumpus>
not that they're any use without the master key
< BlueMatt>
i mean ideally we'd have a clear tag on both so that such software can prompt the user with "found a wallet, please enter passphrase"
< BlueMatt>
but now we're going down a rewrite-wallet-format rabbit hole
< gmaxwell>
jonasschnelli: I don't know how to manage the multiple keys case. One possiblity would be to make the recovery key unencrypted by default, and have an advanced dialog that lets you set encryption for it. And support reading in encrypted ones.
< jonasschnelli>
Yes. That would be great
< gmaxwell>
jonasschnelli: I have a lovely suggestion for hardware wallet friendly KDFs for these things too.
< achow101>
BlueMatt: I propose that we just deprecate the wallet :p
< morcos>
May I make a meta suggestion.. I think we often lose progress on ideas like this by not having someone document what we discussed. could we ask for volunteer every time we have a good discussion like this to draft up a plan.
< luke-jr>
achow101: I get the feeling often
< jtimon>
ack on starting in no-wallet mode
< jonasschnelli>
gmaxwell: +1 (happy to hear)
< achow101>
morcos: meeting notes writer
< achow101>
morcos: he'll write the meeting notes sometime after exams this week
< wumpus>
achow101: and then what, change it into an art project where you can look at blocks drifting by, without being able to do anything? :p
< luke-jr>
wumpus: write a new one :p
< wumpus>
luke-jr: you can do that without deprecating anything
< morcos>
yeah but i mena more a focused thing... like after SF devcore -> plan for Segwit wallet ; this meeting -> plan for wallet encryption recovery code
< wumpus>
morcos: yes, we shouldn't forget segwit wallet
< morcos>
woohoo!
< wumpus>
morcos: that's the thing people are actually waiting for now :)
< sdaftuar>
sipa: thanks!
< gmaxwell>
FWIW, sipa has been working on a stronger base=32 BCH code for things like private keys and stealth addresses; which could be an option for recovery codes.
< wumpus>
sipa: nice!
< luke-jr>
clicking that link won't have permissions for my audio :p
< BlueMatt>
when segwit wallet
< achow101>
sipa: cool!
< achow101>
BlueMatt: soon(tm)
< luke-jr>
(mini rant: using #include <…> for our own files is stupid)
< wumpus>
luke-jr: sigh, the other alternative would have been to fix all relative includes, but that was discussed in detail in the PR and the one before it
< wumpus>
luke-jr: so using #include "../primitive/block.h" in e.g. the wallet. This roots everything at the project root, which is just as unambigious and shorter...
< luke-jr>
I just hope /usr/include/primitive/block.h gets ignored
< wumpus>
that doesn't get ignored either with ""
< luke-jr>
:|
< gmaxwell>
obviously we need to rename every header file to filename_bitcoin_core_is_awesome.h
< wumpus>
at least not the way we were using it, which is essentially as <>, I think if you use "" relatively you can avoid it
< wumpus>
yep!
< luke-jr>
gmaxwell: I'm thinking more of malware infecting builds this way
< wumpus>
well if your build root is infected you're fucked anyway
< jnewbery>
luke-jr: in any case, the PRs were open for a few months (much longer than I would have liked in fact). There was opportunity to comment on those PRs. I think the ship has sailed now.
< luke-jr>
true
< wumpus>
protecting against that is even more questionable than encrypting your wallet, against any possible realistic threat model
< gmaxwell>
luke-jr: well if your host is compromised it's pretty unlikely that it would be limited to only tripping you up with shadowed include files.
< wumpus>
jnewbery: indeed, it's almost as if he waited for it to be merged
< * BlueMatt>
nominates #11363 for cfields' high-priority-for-review, cause he apparently isnt here...
< sipa>
yay c++20... which we'll switch to in 20125?
< sipa>
*2025
< wumpus>
yes, in 20125
< luke-jr>
:x
< Chris_Stewart_5>
ack
< meshcollider>
lol
< jonasschnelli>
heh
< gmaxwell>
change in topic, anyone have recent stats for the number of remaining btc1 nodes-- which are likely about to become a distributed DOS attack on the bitcoin network?
< luke-jr>
(I'm going to drop as soon as the meeting is officially over. I'll be back a few minutes later in case there's stuff to talk about)
< jonasschnelli>
I can filter my seed crawler for uagent string?
< gmaxwell>
meshcollider: ha. I didn't expect them to shut off that fast, I guess they were really almost all just a couple people sybling.
< gmaxwell>
meshcollider: okay, probably not much to worry about.
< meshcollider>
gmaxwell yeah lol there was a Reddit post which went into some detail showing 90% were hosted by AWS
< wumpus>
PSA before the meeting is over: I want to collect corrupted leveldb files, if you have a leveldb corruption please patch https://github.com/bitcoin/bitcoin/pull/11674 and send me the indicated corrupted file.
< jonasschnelli>
861 peers with "Bitcoin ABC" and 100% uptime during last two hours.
< luke-jr>
jonasschnelli: that's just BCH
< achow101>
gmaxwell: my btc1 node is connected to 34 other btc1 nodes, so at least 35
< meshcollider>
ABC is not btc1
< BlueMatt>
i mean its what we did 0.15.1 for, no?
< gmaxwell>
BlueMatt: yes, sure I wanted to know how many there weer because if there were thousands I'd make a post on reddit to urge people to upgrade to 0.15+ seems it might not be needed.
< jonasschnelli>
meshcollider: what uagent does btc1 uses?
< luke-jr>
jonasschnelli: /Satoshi:1.*/
< jonasschnelli>
ah
< achow101>
jonasschnelli: most have a uacomment with "2x"
< jonasschnelli>
107
< gmaxwell>
with only 140ish it's pretty unlikely many nodes will get isolated behind them.
< achow101>
what block were they forking at?
< achow101>
(I need to add it to my site)
< gmaxwell>
494784
< jtimon>
weren't they using a naming just the same as bitcoin core but increasing a version? (ie 0.14.3)
< gmaxwell>
hopefully someone will mine a couple blocks on that fork to help get those nodes disconnected.
< gmaxwell>
jtimon: they made the major version 1.
< achow101>
gmaxwell: a mining pool announced that they would go with the 2x fork regardless
< jtimon>
oh, right
< jnewbery>
won't they disconnect themselves once a valid block is found?
< wumpus>
ding dong
< gmaxwell>
achow101: that was 'bitpico' who is crazy.
< gmaxwell>
it's meaningless.
< achow101>
oh
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Nov 16 20:01:11 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< jonasschnelli>
gmaxwell: I'm happy to hear your bip39 successor HWW KDF idea...
< jonasschnelli>
PBKDF2 with 2048 rounds seems not ideal (BIP39)
< sipa>
jonasschnelli: the idea is a mechanism that allows you to enter the passphrase on a HW device, have the HW device outsource the hardening to a desktop computer (with more power) without revealing the passphrase
< sipa>
and then being able to verify the computer did the hardening correctly
< jonasschnelli>
+1
< sipa>
adam back proposed a scheme for this a while ago, but it's purely CPU dependent
< sipa>
whether it can be combined with memory hard hardening is an open question i think
< jonasschnelli>
purely CPU is still much better then 2048-PBKDF
< sipa>
haha, yes
< jonasschnelli>
Somethine we (HWW company) do discuss regularly is how we can make the backup situation better.. a lot of things are involved. Bip39, sdcard, shamir's secret, notary services, etc.
< jonasschnelli>
I'm not sure if a plain text seed dump (or BIP39) is something you want in a bank tresor
< goatpig>
mdisc?
< jonasschnelli>
mdisc?
< goatpig>
it's basically a cdrom made out of rock
< goatpig>
really really durable
< jcorgan>
i'm not sure if the durability is really demonstrated, but i do have quite a few encrypted live boot images burned to them
< goatpig>
it's hard to demonstrate in practice
< jcorgan>
it's a bit like closed-source software, the media is trade secret, but independent testing was pretty good