< wumpus>
gmaxwell: I like the idea of always adding a warning to getinfo's warning field though. We could introduce that in a minor 0.14 release
< jonasschnelli>
Has there been talk to default enable -walletrbf=1 in 0.15? Or should the direction be that one can set the rbf flag per send*?
< wumpus>
jonasschnelli: not sure about that. We'd have be sure there are no significant wallets or companies that have problems accepting RBF transactions before making it default
< cannon-c>
Default RBF would make fee increase for faster processing easier for the non-tech savvy. I see lots of
< cannon-c>
questions referring how to get transactions unstuck, most wallets that are not core, are difficult for non-technical to resend transaction.
< cannon-c>
But I believe should notify users that RBF is set, with easily accessible ability to uncheck. Just my opinion
< cannon-c>
Wouldn't RBF fee increase in lieu of sending new transaction that is stuck, prevent transaction ID from changing? If so I see how RBF would be great advantage
< gmaxwell>
wumpus: fwiw, I believe several other wallets are producing OO rbf (electrum and green address)
< gmaxwell>
wumpus: also, you can always bump your way out of flagging, which should reduce the issue.
< gmaxwell>
jonasschnelli: I think we should hear feedback for how things go for bumpfee users in 0.14.x
< jonasschnelli>
I think OO rbf should be the default and optionally, if you want to pay a 0-conf merchant, you may want to switch it of.
< jonasschnelli>
*off
< gmaxwell>
jonasschnelli: I think bumping it off is somewhat superior to switching it off, unless you're really sure you want it off.
< gmaxwell>
rationale: even if the rx side will ignore the unconfirmed tx with it off, it might just confirm right away.
< wumpus>
gmaxwell: okay that would be a good argument to enable it by default on master now, at least
< jonasschnelli>
0.14.0rc1 sync against random peers
< jonasschnelli>
2017-02-19 23:06:26 UpdateTip: new best=000000000000000002204d1e9885416bfef39a2259a48a3eb5d36b6e8e21fad7 height=453816 version=0x20000002 log2_work=86.006491 tx=197951144 date='2017-02-19 23:00:41' progress=0.999994 cache=4096.3MiB(7229926tx) warning='2 of last 100 blocks have unexpected version'
< wumpus>
is it syncing or stuck?
< jonasschnelli>
No.. it's fast. :)
< gmaxwell>
453816 is currentish.
< wumpus>
ooh! great.I thought you were worried about the version warning
< jonasschnelli>
453816 was at 2017-02-19 23:06:26
< gmaxwell>
that is with increased dbcache, obviously.
< jonasschnelli>
7GB cache. yes.
< wumpus>
oh only 7GB cache :-)
< gmaxwell>
a little cheating there because it has never flushed at all. :P
< jonasschnelli>
Almost same sync-times then we has with 0.13.0 but more blocks. :)
< wumpus>
yes seems the bottleneck now is really i/o for the utxo database
< gmaxwell>
well, sipa has code that makes it ~33% faster.
< sipa>
last week gmaxwell and i tried to sync a 0.7.2 node from scratch, with -connect to a fast new node
< jonasschnelli>
flush takes only a couple of seconds on that machine.. so neglectable
< gmaxwell>
jonasschnelli: flush of the utxo cache, with 4gb in is not going to take a couple seconds.
< gmaxwell>
try a minute.
< wumpus>
but yes with database in RAM or on a SSD it's surprisingly fast
< jonasschnelli>
I'll measure my shutdown now...
< gmaxwell>
also, part of the time is hidden in the next start up.
< wumpus>
too bad that that doesn't help my odroid64 :-)
< sipa>
jonasschnelli: flushing means that every subsequent spend from an out-of-cache tx afterwards needs a read from disk and deserialization
< jonasschnelli>
Maybe 15s
< jonasschnelli>
I'm using a 1GB/s SSD
< sipa>
so the slowdown is partially in later entries
< wumpus>
yes it's not the flushing itself, as in writing, that poses a bottleneck with leveldb. leveldb is very fast with writing. It's that the flush empties the entire db
< sipa>
half of the time is constructing the batch for leveldb to write
< wumpus>
<gmaxwell> well, sipa has code that makes it ~33% faster.<- that's good news! how?
< sipa>
wumpus: per txout caching
< wumpus>
sipa: ah nice
< gmaxwell>
It unfortunately increases the size of the utxo database on disk somewhat.
< sipa>
and undo data
< wumpus>
by how much?
< gmaxwell>
Less than double. I think it's a clear win.
< gmaxwell>
Basically the keys get repeated.. leveldb has some compaction of that, but its not perfect.
< wumpus>
could anything be gained by compressing undo data?
< gmaxwell>
and the undo data is only slightly larger.
< wumpus>
as you say "repeated", compression seems obvs answer
< sipa>
undo data is already compressed pretty well
< wumpus>
ok
< gmaxwell>
well we can reduce the size of all blocks by ~27% using different stuff, I don't think we've looked to shrink undo data much more than it alread is.
< sipa>
but the size increase on the chainstate should be investigated more carefully
< sipa>
if it increases significantly, it may not be a win for systems with slow disks
< wumpus>
true, undo data is deleted together with the blocks on pruning
< wumpus>
chainstate growth is much more worrying
< sipa>
yes :(
< gmaxwell>
also, I'm sure now we have enough improvements that the sse2-sha256 will be more of a speedup, I think jonasschnelli benchmarked 5% in IBD time, and that was before assumevalid.
< gmaxwell>
wumpus: well the growth, whatever it is, should be roughly a constant factor.
< jonasschnelli>
Yes. IBD is not much of a difference...
< jonasschnelli>
with wumpuses sse2
< wumpus>
5% is pretty nice
< jonasschnelli>
Yes. Sure.
< gmaxwell>
jonasschnelli: I think 5% isn't bad. y'all expect too much from optimizations. And on top of assume valid and that 33% gain that 5% will be much larger.
< sipa>
oh, so... we were unable to get 0.7.2 to sync past some block in 2015
< jonasschnelli>
But I think it's great for nodes running in sync.
< wumpus>
compound growth, heh
< gmaxwell>
even with massively increased locks.
< wumpus>
all those 5% and 3% stacked on top of each other do count
< gmaxwell>
In libsecp256k1 we celebrate 1% improvements.
< jonasschnelli>
We often focus on IBD improvments,... they are great. Boiling hot water quick is great,... but keeping it warm with low amount of energy also counts. :)
< sipa>
well IBD time is a biased proxy for block validation time
< sipa>
and it keeps users happy
< sipa>
many of the improvements to IBD time do matter for single node validation... except assumevalid, though we get sigcache in return
< jonasschnelli>
Yes. That's true. But sse2 5% in IBD may results in 10-15% during in-sync state because of lesser IO. My Pine64 (RPi clone) would really love that.
< cbits>
@jonasschnelli I'm in the camp that rbf should either be set by default with a checkbox to unset it per transaction, or entirely off in the settings. Or the other option, of having it not set by default, but you get the behavior I first described if you set rbf on.
< cbits>
Electrum currently makes you set rbf on for every transaction even after making the option visible in the settings. Which imo is annoying. Should be default.
< jonasschnelli>
cbits: I think the GUI can handle RBF pretty well. I see more potential in improving the RPC API.
< jonasschnelli>
It seems that the prompt breaks the make-base-vm process
< wumpus>
strange
< wumpus>
is this a sudo issue in the VM or the host machine? confusing
< wumpus>
looks something *on* the vm, so changing sudoers won't help you either
< jonasschnelli>
Yes. Seems to be on the VM.
< jonasschnelli>
I'm using make-base-vm on a non-VM host. Using -KVM
< wumpus>
I created a new base image twice yesterday - no issues
< wumpus>
this is kvm or lxc?
< jonasschnelli>
KVM
< wumpus>
okay what I tried is LXC
< jonasschnelli>
Just tried LXC and seems to have worked..
< wumpus>
the upgrader should probably be passed some kind of silent/don't prompt flag for the KVM image build
< jonasschnelli>
After using LXC, the base image was named "base-trusty-amd64" instead of "base-trusty-amd64.qcow2"... renamed and trying to gbuild now
< wumpus>
jonasschnelli: that is the correct naming
< wumpus>
lxc image is not a qcow
< jonasschnelli>
ah... I guess I need to set USE_LXC
< wumpus>
yep
< jonasschnelli>
libexec/config-bootstrap-fixup: line 15: target-bin/bootstrap-fixup.in: No such file or directory
< wumpus>
never seen that one before
< wumpus>
not sure if KVM will work at all in my setup, but will try making a KVM image
< wumpus>
jonasschnelli: I get the same error as you
< bitcoin-git>
bitcoin/master 9fc7f0b Luke Dashjr: Bugfix: RPC/Mining: GBT should return 1 MB sizelimit before segwit activates
< bitcoin-git>
bitcoin/master 279f944 Luke Dashjr: QA: Test GBT size/weight limit values
< bitcoin-git>
bitcoin/master aa791e2 Wladimir J. van der Laan: Merge #9619: Bugfix: RPC/Mining: GBT should return 1 MB sizelimit before segwit activates...
< bitcoin-git>
bitcoin/master 13f6085 Wladimir J. van der Laan: netbase: Make InterruptibleRecv return an error code instead of bool
< bitcoin-git>
bitcoin/master 3ddfe29 Wladimir J. van der Laan: netbase: Do not print an error on connection timeouts through proxy...
< bitcoin-git>
bitcoin/master 7639d38 Wladimir J. van der Laan: Merge #9726: netbase: Do not print an error on connection timeouts through proxy...
< bitcoin-git>
[bitcoin] laanwj closed pull request #9726: netbase: Do not print an error on connection timeouts through proxy (master...2017_02_intr_recv_error) https://github.com/bitcoin/bitcoin/pull/9726
< paveljanik>
sipa, wumpus: #9810 - any idea (seems to be crlf on serialization ;-)?
< wumpus>
that could be the explanation, good catch! how is the file opened?
< wumpus>
if its opened in text (instead of binary) mode, windows will convert \n to \r\n
< paveljanik>
it is written with "w" and opened with "r"
< wumpus>
that should be "wb" and "rb"
< wumpus>
so you've found the issue, congrats
< paveljanik>
Hmm, this reminds me: zerocoin was missing =, we are missing b ;-)
< paveljanik>
we should congrat to the reporter!
< wumpus>
I mean you've found the probable solution
< paveljanik>
but really: any dev with Windows to really test this?
< wumpus>
are you going to make a PR?
< paveljanik>
for master and you'll backport?
< wumpus>
yes
< paveljanik>
give me a few minutes
< bitcoin-git>
[bitcoin] paveljanik opened pull request #9813: Read/write mempool.dat as a binary. (master...20170220_mempool_binary) https://github.com/bitcoin/bitcoin/pull/9813
< wumpus>
paveljanik: thanks
< paveljanik>
you're welcome!
< paveljanik>
we use "w" for PID and log
< paveljanik>
and in tests
< paveljanik>
all "text" files
< cfields>
windows users who ran rc1 will need to delete mempool.dat. Not sure if that
< cfields>
's worth adding to the release notes or not
< paveljanik>
it is auto fixed at the first exit?
< paveljanik>
by saving the new file...
< cfields>
ah, right.
< wumpus>
for text files it makes sense not to add 'b', otherwise you cannot open it in notepad
< achow101>
paveljanik: I can test on windows, as soon as I my windows node finishes syncing
< paveljanik>
achow101, great!
< achow101>
if only we could figure out how to make cross compiling work on xenial... cross compiling for windows is such a pain now
< achow101>
paveljanik: your fix didn't work for me on ubuntu 16.04
< paveljanik>
achow101, on Ubuntu?
< paveljanik>
you have tried the provided mempool.dat?
< paveljanik>
My fix fixes save on Windows...
< achow101>
oh, it was windows specific? I am testing on both ubuntu and windows, just waiting for windows to finish building
< achow101>
paveljanik: for some reason I had to delete the mempool.dats before it would work properly
< paveljanik>
yes, this is probably caused by the issue fixed ;-)
< paveljanik>
achow101, can you please try again with the master-saved mempool.dat?
< paveljanik>
and then run fixed tree with the old mempool.dat?
< paveljanik>
it won't use it. But at exit, it should rewrite it and then on the new start it should use it.
< achow101>
ok, it looks like it does rewrite the old one and actually uses it
< paveljanik>
achow101, perfect!
< BlueMatt>
what am I missing? why are we going out of our way to prefer data() over &v[0]?
< BlueMatt>
I mean its more readable, sure, but I dont believe it is safer in any measurable way?
< BlueMatt>
or, maybe I'm missing in docs...is it that data() is defined for 0-length vectors (though not dereferenceable - it can return any garbage it wants), but &[0] is not?
< gwillen>
BlueMatt: I believe &[0] should be defined for zero-length vectors, it's legal to take a pointer to the slot right after the end of an array (this is what e.g. a .end() iterator is)
< midnightmagic>
huh, that's interesting. I've been fighting with almost that exact same thing.
< BlueMatt>
gwillen: hmm? I mean that doesnt inherintly mean C++ allows it as well?
< sipa>
BlueMatt: vect[0] is a reference to non-existing data if vector is empty, which is illegal
< sipa>
you can have a pointer to non-existing data, but not a reference
< BlueMatt>
ahh, ok, so it is illegal in C++ but not C?
< gwillen>
sipa: but &foo[0] is a pointer, not a reference
< BlueMatt>
(well, vector but not array)
< gwillen>
and a pointer to one past the end is definitely legal
< sipa>
gwillen: &foo[0] = &(foo[0])
< sipa>
that's taking the address of a reference to non-existing data
< BlueMatt>
sipa: see stackoverflow link from gwillen, it sppears to be legal in C
< gwillen>
do you believe it's different in C++ with a vector than in C with an array?
< sipa>
C doesn't have references
< sipa>
the question doesn't apply
< gwillen>
right
< sipa>
a[b] in C is just syntactic sugar for *(a+b)
< gwillen>
but in C that construction would appear to actually dereference an invalid pointer, but the standard gives it a pass
< gwillen>
and explicity says that if you do &foo[x] it turns into foo+x and does not dereference
< sipa>
well that's about arrays
< * gwillen>
nods
< sipa>
which are defined by the language
< BlueMatt>
mmm, ok
< gwillen>
well, it actually also says this pass applies to the * operator as well, and &(*foo) does not dereference either, which is slightly more general
< sipa>
but vector is a standard library class, where vect[x] is an operator that returns a reference to a vector element
< gwillen>
but I suppose that would be hard to spec for user-defined operators
< * gwillen>
nod
< sipa>
which is just impossible to do if there are no elements
< sipa>
for example, a vector implementation (and in practice does) contains a pointer to the first element of the data
< sipa>
but initializes it to nullptr if there are no elements allocated
< sipa>
vect[0] would then be implemented as a dereference of nullptr
< * gwillen>
no
< sipa>
so forcing vect[0] to be valid for empty vectors would outlaw the obviously simplest implementation
< gwillen>
nod*
< BlueMatt>
yea, ok
< sipa>
whereas array[0] is simply a reference to the element 1 past the array... not nullptr
< gwillen>
right
< sipa>
actually, i believe that C and/or C++ require every object to have non-zero allocation space
< sipa>
otherwise alias detection is nearly impossible
< luke-jr>
hmm
< luke-jr>
&foo[0] actually called operator[] in C++, rather than just being (foo+0)?
< sipa>
for vectors, yes
< luke-jr>
I guess it would have to :x
< sipa>
for arrays i don't know
< luke-jr>
a shame, &foo[0] looks nicer XD
< sipa>
also, (foo+0) for a vector would be nonsense... you can't add 0 to a vector