< gmaxwell>
sdaftuar: Do you have any idea if your collected transaction flux dataset has any sigops attacks in it?
< gmaxwell>
sdaftuar: I was thinking it would would be interesting to try some efficient approximations of multidimensional optimization. I have a fairly concrete idea of what I'd like to try.
< gmaxwell>
sdaftuar: e.g. lowering the initial sigops lagrangian and having it increase only when the sigops limit is hit. ... though any change in it would require rescoring the whole mempool, I think that could be done infrequently.
< echelon>
is anyone working on adding support for boost-1.66.0
< gmaxwell>
echelon: what isn't compatible?
< gmaxwell>
We generally prefer to reduce boost dependency, so if there is any incompatiblity that can be resolved by removing boost stuff (and e.g. replacing with C++11 functionality), that would be good.
< jonasschnelli>
What do you mean with " addresses for the pubkey part"?
< gmaxwell>
I would expect the argument you would supply would be addresses, exactly like importing for a watching wallet. Not pubkeys.
< gmaxwell>
(I suppose a pubkey would be okay to accept in addition to addresses, but I'm not sure if anyone would use it)
< jonasschnelli>
Using addresses would mean, you would have to known what script types where used for the unspents you are looking for...
< sipa>
how would you not know?
< gmaxwell>
the address tells you the script type. Any address can be converted into a scriptpubkey, otherwise how could someone pay it?
< gmaxwell>
(also, with a bare pubkey you're left guessing at the script type)
< sipa>
maybe this is easier to discuss with explicit use cases in mind?
< jonasschnelli>
Yes... hard to give arguments against addresses,...
< gmaxwell>
The use case is in the linked PR.
< jonasschnelli>
my use case is that there are backups where only pubkey/privkeys are visible
< jonasschnelli>
But could also be addresses...
< jonasschnelli>
I though pubkeys and derive all common scipts makes it easier ...
< gmaxwell>
The only shortcomming of also supporting pubkeys is that pubkeys don't encode the script type; and we are trying to move away from the "use all the types" handling in the wallet in general.
< sipa>
well i think it's a bad situation we're in now that public keys explicitly don't have an associated script type
< sipa>
which simplifies things "they can be anything" in one way
< jonasschnelli>
Maybe both should work
< sipa>
unfortunately at this point we really need to treat xpubs as deriving all address types that currentpy exist
< jonasschnelli>
sipa: that's why you want to scan for all possible related funds...
< gmaxwell>
I would say that it's fine to also support pubkeys and do wildcard handling, but I'm a little concerned that it'll make it harder to rip out wildcard handling in general in the future.
< gmaxwell>
jonasschnelli: that has pretty poor scalablity long term as the number of possible types increases.
< sipa>
jonasschnelli: yeah, compatibiloty with how things are done now is a reason in favor
< gmaxwell>
at least for the general usage in the wallet, perhaps it's excusable in an import.
< sipa>
compatibiloty with how i hope thongs work in the future is an argument against
< sipa>
*some s/o/i/ where appropriate
< jonasschnelli>
I agree... but I guess the only point where we want that fallback-scan-for-everything is exactly the utxo-base-sweep (scan)
< jonasschnelli>
(or at least an option)
< sipa>
fair
< sipa>
i agree
< jonasschnelli>
I think supporting addresses and pubkeys makes most sense... will add
< sipa>
it's analogous to importpubkey and importprivkey
< sipa>
which also derive everything
< jonasschnelli>
Right
< sipa>
or at least all existing addresses
< gmaxwell>
I think for sweeps its probably fine. Addresses should be the recommended input, if you have a choice, and they don't have that problem. Later we can introduce a script type augmented xpub to cover that case too.
< jonasschnelli>
There is also a missing rpc call for a rawsweep... something that can calculate fees based on an array of unspents and a send-to-address
< sipa>
longer term i hope we have address-type-specific privkeys/xpubs (like electrum has)
< gmaxwell>
I think the behavior should be analogous to importaddress-- e.g. equal or a superset of what that can import.
< gmaxwell>
jonasschnelli: can fundrawtransaction do that already?
< gmaxwell>
without thinking too hard I think fundraw could be made to do that, without breaking compatibility with anything.
< sipa>
yes createraw + fun draw can donthat, but it's abit cumbersome
< jonasschnelli>
gmaxwell: I don't think so it can today... what you want is no change outputs, just the total - the required fee after the given confirmation target
< sipa>
*do that
< gmaxwell>
sipa: if the app is still unclear, consider e.g. sweeping spinoff coins without having to do a importaddress and wallet rescan on a spinoff node.
< jonasschnelli>
fund* is also the wrong term if you have all unspents
< sipa>
right
< sipa>
yes, makes sense
< gmaxwell>
jonasschnelli: well thats the change from payment amount behavior.
< jonasschnelli>
What would easy work is allowing an send-to-address in the utxo scan and spit out the recommended fee
< jonasschnelli>
(for the tx in the background and use the estimator)
< jonasschnelli>
but meh
< gmaxwell>
I suppose someone could then edit the tx if they want other behavior.
< gmaxwell>
ah, I see ... maybe this should wait for achow's PSBT support.
< jonasschnelli>
maybe the scantxout command should spit out (additionally) the raw tx for a sweep to a given address
< gmaxwell>
The best interface for this, I think, would be to output a raw transaction. But the problem with that is that it isn't an acceptable input for signrawtransaction.
< gmaxwell>
I suppose it could output BOTH a raw transaction and the argument for signrawtransaction.
< jonasschnelli>
Yes.
< gmaxwell>
But that problem is the primary motivation for PSBTs.
< jonasschnelli>
Though we would need xpriv support in signraw
< gmaxwell>
well one of the primary motivations.
< jonasschnelli>
(can be added at later stage=
< gmaxwell>
I like it outputting the raw transaction and signraw argument, though a PSBT would be even better.
< gmaxwell>
it could take a destition array, and if it's empty, then just leave off the destinations for further editing.
< jonasschnelli>
gmaxwell: I think it does already output the signraw arguments,.. the "unspents" array can be used for singraw (it contains the scriptPubKey)
< gmaxwell>
and support a feerate argument override for the estimator as some of the other rpcs do.
< jonasschnelli>
(but not the rawtx)
< gmaxwell>
I know.
< jonasschnelli>
gmaxwell: good point, feerate override would be great to have
< gmaxwell>
esp because estimator is dumb on recently started nodes... and I think for this a common use will be to start something, catch up to tip, and sweep...
< jonasschnelli>
Oh.... yes.
< jonasschnelli>
I hope we have disabled the default fallbackfee for such cases...
< gmaxwell>
listunspent has a modifier dictionary that it takes, which you might consider having. The obvious modification would be to ignore dust outputs.
< gmaxwell>
(other modifier might be height<=x, which could _sometimes_ be used to create spinoff txn without the spinoff chain-- though that could be added later)
< jonasschnelli>
Great ideas...
< gmaxwell>
e.g. you have a signer that knows how to sign for SBTC, and you haven't spent any coins since the fork.. you could use a bitcoin node to create the sweep transactions...
< jonasschnelli>
One downside of those utxo scans is: "real 8m38.318s"... they take relatively long,... 8m is on a very fast SSD machine
< gmaxwell>
that is still a HELL of a lot better than a full wallet rescan.
< jonasschnelli>
indeed...
< jonasschnelli>
Also,... pruned peers
< gmaxwell>
8m still seems like a long time, our poor poor utxo set.
< gmaxwell>
oh I assume you flush first, perhaps some of that was flush time?
< gmaxwell>
which can take a long time if your dbcache is gigantic.
< gmaxwell>
hm. I seem to recall the gettxoutsetinfo taking ~2 minutes. :(
< jonasschnelli>
gmaxwell: I removed the flush...
< gmaxwell>
uh, but if you don't flush you won't return correct results?
< gmaxwell>
flushing is needed to make the utxo set on disk consistent with the chaintip.
< sipa>
unless you scan both the disk and memory explicitly
< gmaxwell>
I suppose you can do that... scan disk, then memory to remove anything found but spent, and add anything added... seems complicated.
< jonasschnelli>
I guess I should re-add that. :)
< sipa>
i believe ryanofski has a PR to pet you scan through a view of the merged disk+cache
< sipa>
*let
< jonasschnelli>
8min is on a AMD Ryzen 7 1700X, 6Gb/s SSD (software raid though)
< echelon>
gmaxwell: off-topic, but did you used to do ports for zipit?
< gmaxwell>
echelon: I don't know what 'ports for zipit' would be, so presumably not. :)
< echelon>
zipit z2, the handheld linux console thingy
< echelon>
nevermind :)
< gmaxwell>
nope! only handheld linux things I've done anything with are sharp zarus and openmoko.
< echelon>
kk, gotcha
< gmaxwell>
jonasschnelli: just as a sanity check, can you time a gettxoutsetinfo on the same host?
< jonasschnelli>
gmaxwell: one sec
< gmaxwell>
it might be that my cached memory of how long that takes is outdated or just wrong... but if not, there might be something to optimize in your code.
< jonasschnelli>
Oh... it's --enable-debug
< sipa>
jonasschnelli: don't do that when benchmarking :)
< jonasschnelli>
I fall into this over and over again
< sipa>
i'm happy sokeone is regularly running with debug enabled :)
< sipa>
*someone
< gmaxwell>
lol
< gmaxwell>
whew, well I hope its faster without it.
< gmaxwell>
not that 2m would be all that awesome either.
< jonasschnelli>
real 9m38.249s (gettxoutsetinfo)
< jonasschnelli>
now trying with no-debug
< jonasschnelli>
oh... much faster
< gmaxwell>
okay so under a minute
< jonasschnelli>
UTXO scan: real 0m47.572s
< gmaxwell>
whew
< gmaxwell>
now to just figure out how to get future spinoffs to base themselves off 0.17 or whatever will have this feature. :P
< wumpus>
jonasschnelli: I've also fallen into that trap a few times; maybe it'd make sense to make --enable-debug builds emit a warning in the log, as well as when running the benchmarks
< gmaxwell>
maybe add an extra colum "DEBUG" after the timestamp in all log output?
< gmaxwell>
or something like that.
< gmaxwell>
though I bet jonas was doing time ./bitcoin-cli gettxoutsetinfo :P
< wumpus>
hehe that's very intrusive :)
< wumpus>
on every log line
< gmaxwell>
I dunno about you, though, if it only logged it at start I'd never notice. I've run the wrong binaries many times and missed the logged version numbers.
< wumpus>
that veers into the domain of actively discouraging people to run debug builds
< gmaxwell>
but fair enough.
< gmaxwell>
it would be less intrusive if we already had a column there.
< wumpus>
adding just a 'D' or so would be a compromise
< gmaxwell>
like "bitcoin-mainnet"
< gmaxwell>
or B for bitcoin mainnet... so at least your log parsing commands don't break.
< gmaxwell>
ah interesting, we each thought it was intrusive for different reasons.
< wumpus>
but in any case, logging it in some way is better than not logging it, even if it's just an adition to the version message, you can always check/correlate logs later
< gmaxwell>
true. +1 for at least sticking it in the version string that gets logged.
< wumpus>
I learned to always check the version message in the log painfully by bisecting kernels on embedded devices; it happens that the programming did go well, so I ended up testing the wrong kernel version, making me bisect 15 steps over again :p
< wumpus>
did not*
< wumpus>
yes it might also be intrusive for another reason (e.g. breaking log parsers)
< bitcoin-git>
[bitcoin] laanwj opened pull request #12197: Log debug build status and warn when running benchmarks (master...2018_01_debug_in_log) https://github.com/bitcoin/bitcoin/pull/12197
< bitcoin-git>
[bitcoin] laanwj opened pull request #12198: rpc: Add deprecation error for `getinfo` (master...2018_01_getinfo_deprecation_message) https://github.com/bitcoin/bitcoin/pull/12198
< provoostenator>
Functional test suite passes on my machine a lot more often these days, so kudos to whoever improved things.
< provoostenator>
Now I just need to find out how to prevent those dozen firewall permission popups on OSX after each recompile.
< wumpus>
you get firewall permission popups for localhost?
< aj>
wumpus: for #11774, i figure it makes sense to wait until #12101 and #12119 (and any other 0.16 PRs modifying tests?) get merged... should i rebase it so github is happy, or does merging it via git directly work okay?
< gribble>
https://github.com/bitcoin/bitcoin/issues/12101 | Clamp walletpassphrase timeout to 2^30 seconds and check its bounds by achow101 · Pull Request #12101 · bitcoin/bitcoin · GitHub
< gribble>
https://github.com/bitcoin/bitcoin/issues/12119 | [wallet] use P2WPKH change output if any destination is P2WPKH or P2SH-P2WSH by Sjors · Pull Request #12119 · bitcoin/bitcoin · GitHub
< wumpus>
aj: you'd need to rebase it just before merging, I guess
< wumpus>
after the PRs you're waiting for go in, at least
< wumpus>
rebasing *now* isn't worth it if you just have to do it again
< aj>
wumpus: rebasing just means running "./fix_tests" :) (well, and eyeballing the patches, and checking the tests actually run)
< provoostenator>
wumpus: it seems so, haven't investigated what exactly these functional test bitcoind nodes are asking permission for
< provoostenator>
wumpus: shouldn't #11536 be in he high priority review list, given that #7729 can be based on it?
< wumpus>
provoostenator: well all the tests can be run without any external network access, so rejecting them to do that shouldn't make a difference, just localhost counts
< gmaxwell>
it might be due to spurrious dns lookups it doing something that looks like network access, I suppose.
< provoostenator>
Right, but I'd rather not have to see that popup at all.
< gmaxwell>
though we'd probably want to fix that if reasonably possible
< wumpus>
gmaxwell: yep, it's fully possible that there is some 'leakage', there, but blocking it shouldn't affect the test results
< provoostenator>
Let me try rejecting...
< provoostenator>
I grew up with Windows as a kid, so I'm used to clicking Yes
< wumpus>
I've never tried runnign the tests on host with only localhost network interface, that'd be kind of interesting (but I don't see why it wouldn't pass)
< gmaxwell>
aj: uh, that sounds bad
< gmaxwell>
oh right due to changing where segwit activated.
< aj>
gmaxwell: 0.15's segwit activations is later than 0.16's, so erroring out is reasonable, assertion failure is just an unpleasant way of doing it
< gmaxwell>
okay not bad... just ugly.
< aj>
yeah
< wumpus>
if it (for sure) only affects regtest it shouldn't block 0.16.0
< gmaxwell>
A trivial deuglyfication could be to adjust the assert with a note about why it's triggering.
< wumpus>
<wumpus> I've never tried runnign the tests on host with only localhost network interface, that'd be kind of interesting (but I don't see why it wouldn't pass) <- it's trivial to simulate that using linux netns, will try it
< gmaxwell>
(via the string equality assert hack)
< wumpus>
yes, that'd make sense
< wumpus>
phew, both functional and unit tests pass with only loopback networking
< wumpus>
theoretically, with the UNIX sockets patches it could work even without a loopback interface, but I'm not going to try that :)
< provoostenator>
Tests seem to be fine if I deny permission. Still trying to see what triggers the permission popup. I assume it's related to what it tries to listen on.
< bitcoin-git>
bitcoin/master 5f911c5 mruddy: trivial: fix address_type help text of getnewaddress and getrawchangeaddress
< bitcoin-git>
bitcoin/master cad504b MarcoFalke: Merge #12177: trivial: fix address_type help text of getnewaddress and getrawchangeaddress...
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #12177: trivial: fix address_type help text of getnewaddress and getrawchangeaddress (master...trivial1) https://github.com/bitcoin/bitcoin/pull/12177
< provoostenator>
OSX remembers the answer to the popup question until you recompile bitcoind with some change (whitespace is enough). src/bitcoind -regtest is enough to trigger it. -listen=0 prevents it
< provoostenator>
-bind=127.0.0.1 also prevents the popup
< wumpus>
right, it probably remembers the hash of the executable. Which is cleverer than some windows 'firewall' software I've encountered which just looks at the name of a program...
< wumpus>
provoostenator: binding to 127.0.0.1 would make sense for the functional test suite
< provoostenator>
Adding -bind=127.0.0.1 to each test node in address_types.py make the popup go away while tests still pass, so that's probably the solution.
< wumpus>
there is no need for the tests to blnd globally, it could even be somewhat of a security risk
< gmaxwell>
I was about to say the same thing (security risk)
< provoostenator>
wumpus: not sure if it's just the hash. Adding whitespace, recompiling and then removing whitespace and recompiling also triggers the popup. But deleting the executable and then running make again doesn't.
< gmaxwell>
like if some future change/mistake resulted in setting static rpc credentials in some test, and enabling remote connections... then someone could, if not worse, at least clobber files you can write to with wallet backups and dumps.
< gmaxwell>
provoostenator: timestamp is pretty cheap to include in a hash... or maybe OSX has some kind of privledged capabilities for executables.
< gmaxwell>
would be pretty useful for AV/backups to be able to set a flag on a file that only the setting program can access, that gets nuked if the file is modified.
< dx25_>
With my update to boost 1.66.0-1, I'm getting this really hard-to-understand (for me) compile error. i guess something should be const that isn't?