< gmaxwell>
cfields: I'd like to see the vla warning go in ( #9789 ) -- but it really should be a warning, as we just encountered a fails in one compiler but not another.
< gmaxwell>
BlueMatt: We're making too many changes with the network without good data on their improvements. The recent ones have been such obviously clear wins that it's okay-- but if we do not get good ways of benchmarking these things running constantly, we're going to eventually make dumb decisions. As a bonus, we'll be able to describe our improvements in ways more specific than mega-super-amazing. :)
< cfields>
gmaxwell: ok. i thought on that a good bit after you raised the objection, and ultimately i agree
< cfields>
gmaxwell: speaking of which, i'm futzing with gnuplot atm :)
< BlueMatt>
gmaxwell: talk to jnewbery :)
< BlueMatt>
(he's working on using the simulation framework stuff to build benchmark CI, and I told him he should set up mini-networks with simulated latency and benchmark that, too)
< cfields>
gmaxwell: plotting out a few different data sets wrt ibd sync between 0.13/0.14
< gmaxwell>
cfields: sweet.
< gmaxwell>
title "Mega-super-amazing performance"
< cfields>
i got tired of arguing with gnuplot over the x axis :\
< cfields>
(it's linear)
< gmaxwell>
set xdata time
< gmaxwell>
set timefmt "%s" ?
< cfields>
gmaxwell: it's real timestamps, not scaled down to 0. So it wants to show crazy-high second-values
< gmaxwell>
it's a neat graph though it really hides a lot of the other improvements on 0.14 since it only goes to 295k
< sipa>
cfields: so subtract the lowest timestamp?
< gmaxwell>
using 1:($2-constant) with lines
< cfields>
gmaxwell: well the 295k is where the checkpoint kicks in in 0.14. I'd prefer to see a separate graph syncing blocks 0 to 100,000 or so without checkpoints/assumevalid
< cfields>
*in 0.13, sorry
< gmaxwell>
it's not just that, the behavior of the blockchain is quite different in the last two years than the first two. :)
< sipa>
i think it's a neat graph
< sipa>
it doesn't show everything, but it's hard to put everything in one graph without misrepresenting one thing or another
< cfields>
gmaxwell: very much so. Clearly 0.14 benefits most from the improved short bursts in the early blocks
< cfields>
but yes, i'd like to do another where validation is forced for all
< gmaxwell>
It would be nice to have a general figure reflecting user expirence for the release note. "With default settings syncing has been reduced from X hours to Y hours on a quad-core system from another local 0.14 node."
< cfields>
gmaxwell: agreed. I have figures for 0.14->0.14 on an ec2 4xcpu, 8gb ram. It's ~3hrs. 0.13.2->0.13.2 was taking too long and i got impatent, wanting to run other tests :(
< cfields>
i can fire that one up again tomorrow and just let it go
< cfields>
(that was dbcache=4000, blocksonly=1, listen=0, disablewallet=1)
< gmaxwell>
ec2 is really IO starved normally no?
< cfields>
*impatient. That was a dangerous typo :)
< cfields>
gmaxwell: unsure. I figured it was representative of how lots of nodes are run, so good data point
< cfields>
no clue how true that is
< gmaxwell>
I'm pretty sure any customization of dbcache limits it to representing <5% of nodes.
< cfields>
mm, good point
< jeremyrubin>
Hey I have an idea...
< jeremyrubin>
what if during validating sync
< jeremyrubin>
hmmm nevermind... let me read more
< * luke-jr>
discovers GCC 4.9 doesn't actually support all C++11 >_<
< cfields>
luke-jr: atomic shared_ptr ?
< luke-jr>
cfields: std::align
< cfields>
huh, i wasn't aware of std::align. Only alignof/alignas
< gmaxwell>
... idiot fake nodes, I have one connected to me that continually sends verack messages instead of pongs.
< sipa>
lol
< luke-jr>
XD
< sipa>
Ḧello! speed v 70012?- sure, i can speak 70012 - have you heard about tx ab315fa5118? - sure i can speak 70012!
< gmaxwell>
"It says ack, right in the name."
< cfields>
heh
< luke-jr>
gmaxwell: verack
< cfields>
makes sense, i think. From the laziest possible implementor's perspective
< cfields>
pretty sure we never penalized dupe veracks
< sipa>
i thought we did
< cfields>
so if you don't feel like coding an extra 3 lines, just always reply with verack. ship it!
< gmaxwell>
yea, if your goal is just to record inv traffic; 'anytime you get anything send this canned verack' is the absolute minimum to make the protocol go.
< luke-jr>
gmaxwell: why wait until you get something? XD
< cfields>
sipa: doesn't look like it. Unless there's something subtle
< sipa>
we do penalize duplicate version, right?
< cfields>
both, now
< gmaxwell>
I don't think we peanalize duplicate verack now, do we?
< luke-jr>
gmaxwell: verack
< cfields>
hmm, i guess not. Could've sworn that went in
< gmaxwell>
I vaguely recall from some other project that there was a reason to not do that ^. ... probably on obscure platforms we don't support.
< gmaxwell>
hm. perhaps I remember incorrectly.
< jonasschnelli>
Still having troubles setting up gitian LXC. The LXC container can't connect to the internet. I don't have a VM for gitian. I run gbuild directly on a host. The host has a single public IPv4 addr. I'm not sure how I can setup the bridging..
< wumpus>
gmaxwell: that's pretty late to start arguing against it we've already done it for 99% of the project
< wumpus>
jonasschnelli: strange.
< wumpus>
jonasschnelli: did you set the GITIAN_HOST_IP and LXC_GUEST_IP?
< fanquake>
wumpus I agree with your comments re gitian building and LXC. That is what I've always used, these problems are all that I've ever had with it. I think the gitian build guide also does a, lengthy, but good job of explaining setup.
< fanquake>
That's using a debian & virtualbox on osx.
< wumpus>
fanquake: thanks. we should just try to fix the guide, at least the parts people stumbling on. Writing another one for KVM, which proabably gets outdated just as quickly isn't much of a solution.
< fanquake>
cfields If I start a node with those params, and quit after some syncing I get a segfault :o Investigating now.
< wumpus>
also I think LXC is conceptually the right solution for this problem - what you want for building is an isolated container, not a full emulated operating system
< fanquake>
wumpus yea, the guide could probably use some improving. I've tried to keep it up to date every now and then. I though we also had a gitian-building script that was meant to automate some of the setting up?
< gmaxwell>
wumpus: fair enough!
< wumpus>
fanquake: I'm kind of disappointed that the person adding the gitian-building script didn't update the guide to use it
< wumpus>
fanquake: that kind of guarantees no one will use his script
< fanquake>
We also link to that Docker setup. I checked the other day and it hasn't been updated in > 1yr. Think we could probably drop that link from the guide.
< wumpus>
gmaxwell: according to the manpage it's even more portable "The use of EXIT_SUCCESS and EXIT_FAILURE is slightly more portable (to non-UNIX environments) than the use of 0 and some nonzero value like 1 or -1. In particular, VMS uses a different convention." VMS!
< luke-jr>
wumpus: I think UML is a better solution :p
< wumpus>
luke-jr: the 90's called and want UML back? :p
< gmaxwell>
maybe thats what I'm remembering! :P
< luke-jr>
☺
< wumpus>
luke-jr: more seriously, I haven't heard of anyone using UML in 10 years or so
< jonasschnelli>
wumpus: I have set GITIAN_HOST_IP and LXC_GUEST_IP to the same public ipv4 addr. I'm not aware of any other IP address that could be used?!
< jonasschnelli>
There is no LAN in my case
< fanquake>
Can anyone confirm this. src/qt/bitcoin-qt --reindex --dbacache=8192 --par=10 --blocksonly=1 --listen=0 --disablewallet=1 quiting after any amount of time gives a segfault.
< luke-jr>
wumpus: I'm surprised you remembered it
< luke-jr>
well, slightly.
< wumpus>
luke-jr: I remember a lot of useless things
< wumpus>
jonasschnelli:i: if you set them to the same IP it will fail. The point is that the host and guest communicate with each other on this bridge network
< jonasschnelli>
Hmm.. maybe I use ifconfig br0 10.0.3.2/24 up for the bridge and set 10.0.3.5 as guest IP and my public ip as host... let me try
< wumpus>
jonasschnelli: so the sequence there is : set up a br0 network, assign ourlelves an IP on it and bring it up, set up masquerading for the interface to the outside, set up forwarding, then tell gitian-builder these things by the way of environment variables
< wumpus>
GITIAN_HOST_IP should be the IP we assign ourselves using ifconfig br0
< wumpus>
LXC_GUEST_IP is the IP that the guest should have, and should be within this subnet
< fanquake>
Ok, the problem is just with —disablewallet=1 , running with that and then quitting will segfault on master.
< jonasschnelli>
Okay. Then GITIAN_HOST_IP=10.0.3.2 and LXC_GUEST_IP=10.0.3.5 should work... ok let me try. I think i also enabled Ipv4 forwarding
< jonasschnelli>
fanquake: Thanks for the report. Is there already an issue on github for that?
< jonasschnelli>
I can try to fix this
< fanquake>
jonasschnelli checking, if not ill open with some details.
< * jonasschnelli>
think he should learn the basics of net bridging...
< jonasschnelli>
Hmm... now I see in build.log that Fetching ccache-3.3.3.tar.bz2 has stalled (no progress at all). I guess this means the target can't connect to the internet?
< fanquake>
Don't you have to clone the repositories first?
< jonasschnelli>
fanquake. I think the nat is not working correctly..
< jonasschnelli>
Can I somehow start the LXC container manually and execute commands in the container? lxc-ls returns an empy list
< luke-jr>
lxc-execute? :p
< jonasschnelli>
luke-jr: lxc-execute: missing container name? And lxc-ls doesn't show any container...
< jonasschnelli>
Found it out. LXC_ARCH=amd64 LXC_SUITE=trusty on-target -u root ping 8.8.8.8
< jonasschnelli>
Ping to the inet from within the lxc container now worked after re-setting up the bridge with 10.0.2.2 (instead of 10.0.3.2)
< jonasschnelli>
hm... dns resolving does not work in the LXC container. That's the issue.
< jonasschnelli>
Ah. I think there is a dnsmasq conflict with the bitcoin dns-seeder
< wumpus>
yes AFAIK it uses the host as DNS server
< wumpus>
it should be possible to make the client use a different DNS server
< fanquake>
You get a similar problem was running with —reindex, in that number of blocks is unknown, but both progress increase and estimated time left are both "calculating".
< fanquake>
So you basically get no gui output expect for the progress bar, and when you hover over it.
< fanquake>
Even the debug window doesn't update with progress.
< fanquake>
Seems less than ideal, and user forced to reindex will have no idea of progress, or how long it's going to take.
< fanquake>
*any
< wumpus>
if it's another lock contention issue it's less than trivial to solve
< wumpus>
so I would prefer not to hold up 0.14.0 for that, no
< fanquake>
Something for 0.14.1 then? I'm sure the complaints of "reindexing with no progress output!" "How long is it going to take!" will roll in.
< wumpus>
but if someone comes up with a solution it could be backported to 0.14
< wumpus>
sure
< fanquake>
I'll update the above issue with some more info, and create a 0.14.1 milestone.
< wumpus>
it'd nice to not have the problem, but reindexing GUI output just isn't critical, in the way that say a crash is
< * luke-jr>
stabs QTableView for not letting him resize the freaking columns
< wumpus>
luke-jr: you should be able to resize the columns?
< luke-jr>
wumpus: tried 3 or 4 different ways, none seem to work :/
< wumpus>
luke-jr: we resize tables in various parts of the source, don't know by heart how though
< luke-jr>
yeah, I must be missing something stupid
< wumpus>
the most tricky thing is not the resizing itself
< luke-jr>
?
< wumpus>
but *how large* to make the columns
< luke-jr>
QFontMetrics::width?
< wumpus>
font sizes can differ, high-dpi versus normal-dpi columns, and so on
< wumpus>
yeah it's possible to get it right, just tricky
< bitcoin-git>
bitcoin/0.14 e22c067 kirit93: Fix for issue #9775. Added check for open() returning a NULL pointer....
< bitcoin-git>
[bitcoin] jonasschnelli opened pull request #9817: Fix segfault crash when shutdown the GUI in disablewallet mode (master...2017/02/sf_gui) https://github.com/bitcoin/bitcoin/pull/9817
< jonasschnelli>
wumpus: re your offline build mode in gitian: how does it can get the depends/packages in the offline mode?
< bitcoin-git>
[bitcoin] ryanofsky opened pull request #9818: Save watch only key timestamps when reimporting keys (master...pr/multirewatch) https://github.com/bitcoin/bitcoin/pull/9818
< bitcoin-git>
[bitcoin] morcos opened pull request #9819: Remove harmless read of unusued priority estimates (master...removeExtraRead) https://github.com/bitcoin/bitcoin/pull/9819
< morcos>
Just a heads up that pruning.py is broken in 0.14 and master, cause seems to be the 2 hour buffer added in #9778. I think ryanofsky is working on a fix.
< bitcoin-git>
[bitcoin] ryanofsky opened pull request #9820: Fix pruning test broken by 2 hour manual prune window (master...pr/prunewind) https://github.com/bitcoin/bitcoin/pull/9820
< morcos>
Question re fee estimation improvements. We have two rpc calls now. estimatefee and estimatesmartfee. I'd like to improve estimatesmartfee to be even "more smarter" and perhaps return more data to you about how it arrived at its estimates.
< morcos>
And I'd like to turn estimatefee into more of a raw call that allows you to generate your own smart estimation through repeated calls to estimatefee with various parameters
< morcos>
but that would imply re-enabling estimates for a target of 1 in the estimatefee call. does anyone think that's necessarily an issue. seems a bit odd to disable for a release and then reenable, but it becomes more of a raw utility, it makes sense to not constrain it
< morcos>
does this make sense as a way forward for improving and extending fee estimation?
< morcos>
or would it perhaps make sense to make a new rpc call for low level functionality that is even more raw and could give you direct acess to the saved data?
< ryanofsky>
imo, disabling and reenabling is nicer than disabling and then adding a new rpc to do what the disabled thing did
< gmaxwell>
funny, I was going to say the opposite!
< gmaxwell>
the purpose of it is changing, so perhaps it should have a new name. "feeestimationdata"
< ryanofsky>
yeah, it's a tradeoff. personally when i am a user of an api, i just want it to be as straightforward as possible what the various options in the api do, and what the behavioral differences are
< ryanofsky>
i care a lot more about the actual behavior of the api, than whatever the intended "purpose" of the api was when it was developed
< morcos>
after some discussion i'm leaning towards this: adding a new hidden rpc call estimaterawfee that allows raw access to the model data. It can be useful for debugging people with fee estimate problems or for super advanced users that want to implement their own smartness
< morcos>
estimatefee itself will continue as is to hopefully eventually be deprecated
< morcos>
and most people will use estimatesmartfee which may return more information to help people who want to read more than one number understand the tradeoffs.. but is more user friendly and doesn't take a whole lot of parameters, just how quickly you want the damn tx confirmed
< bitcoin-git>
[bitcoin] laanwj opened pull request #9821: util: Specific GetOSRandom for Linux/FreeBSD/OpenBSD (master...2017_02_osrandom) https://github.com/bitcoin/bitcoin/pull/9821