< Usman_Mutawakil>
Is their any documentation that expands on the details of the source code? I'm having difficulty finding the entry points.
< Usman_Mutawakil>
there*
< meshcollider>
Usman_Mutawakil: The entry points of the programs, or the entry point to understanding it?
< Usman_Mutawakil>
Both, initially the former
< meshcollider>
Usman_Mutawakil: the main() function for bitcoind is in bitcoind.cpp, for bitcoin-qt it is in qt/bitcoin.cpp and there is one in bitcoin-tx.cpp and bitcoin-cli.cpp for their respective binaries too
< BGL>
can anyone help me with instructions on storing the blockchain &/or the wallet seperately on windows, via symbolic link, drive/mount or anything that works?
< meshcollider>
BGL: are you using 0.15.1 or build of master?
< BGL>
yeah newest copy
< BGL>
0.15.1
< jonasschnelli>
cfields: ping (in case you are still awake)
< cfields>
jonasschnelli: pong, hi
< jonasschnelli>
hi
< cfields>
jonasschnelli: any chance you can get ahold of a dummy certificate signing request?
< jonasschnelli>
cfields: I have one here... one sec
< cfields>
oh, that was easy :)
< jonasschnelli>
cfields: there is now also the option to request ECC (up to 521 bits... though not sure if they will be accepted
< cfields>
ah, nice
< cfields>
we'd need to check back-compat too, though
< cfields>
oh, i guess that's what you meant :)
< jonasschnelli>
cfields: Yes. Not sure if its accepted and backward compat... so better to go for RSA
< cfields>
got the dummies, thanks a bunch. It's late now, but I'll have a look in the morning.
< jonasschnelli>
cfields: Sure. Just tell me if you want access to the apple developer programm interface
< cfields>
jonasschnelli: hmm, we probably need to keep that locked down. assuming that access would allow me to revoke the current or request a new cert.
< jonasschnelli>
cfields: I don't know how to best deal with that...
< cfields>
jonasschnelli: i wonder if it's possible to cancel your account without revoking the certs. a quick search turned up an email address to use for revoking without an account, so maybe that's possible?
< cfields>
then a new account could be created for each renewal. Or would that require re-certification?
< jonasschnelli>
cfields: the account setup is somehow cumbersome. I had to verify the account (by phone) because it's an organisation. You need a DUNS number, and need to pay 99USD.
< cfields>
ah damn, ok. I was thinking the dev account and corp id might not be 100% tied together.
< jonasschnelli>
cfields: and finally, it's social hackable. If we lock ourselfs out,.. apple probably has a solution to lock in again if the right person from the organisation authorizes himself
< jonasschnelli>
cfields: I can setup an new "team member". Maybe I can even revoke my own admin user (after have setup a new one)... but not sure how this would allow to improve things
< cfields>
yea, maybe not
< cfields>
well, i guess the same threshold scheme would work for revocation as well. if ever necessary.
< cfields>
ah nm, it'd be a different key
< cfields>
(assuming someone got control of the account and created a new key)
< cfields>
blah, I'll dive in tomorrow. thanks again :)
< jonasschnelli>
cfields: Yeah. Have some rest. I think it's not solvable in a secure way. On top, apple has control over everything. It's just code signing and does not ensure the binary has built properly. It's far less valuable then the gitian signatures...
< warren>
A while ago checkpoints were there in part to allow IBD to go a lot faster. The checkpoints are still in there now. After headersfirst, are those checkpoints in there still serving any IBD performance purpose?
< sipa>
warren: no, we have assumevalid for that now
< sipa>
the checkpoints are just there to prevent a low-work headers spam attack
< warren>
thx.
< sipa>
assumevalid was introduced in 0.13 or 0.14 iirc
< GAit>
wumpus: is the ARM assembly optimization the one enabled by --enable-experimental-asm or there's something ARM specific that I missed?
< sipa>
GAit: the libsecp256k1 ARM assembly? i don't think so
< GAit>
I was just going on about a comment from wumpus above about considering enabling by default the ARM assembly opt and I was wondering which flag would do that manually today
< GAit>
maybe it's just me but i find having both --enable-experimental-asm and having --with-asm=arm --enable-experimetal is a tiny bit confusing
< meshcollider>
Yeah GAit wumpus mentioned earlier today that some documentation is probably needed but he's not sure where the best place to put it is
< sipa>
GAit: "--with-asm=arm --enable-experimental" are libsecp256k1 configure options; the " --enable-experimental-asm" is a bitcoin core configure option
< sipa>
together it does look confusing, i agree
< wumpus>
--with-asm=arm enables the secp256k1 arm assembly, --enable-experimental-asm does nothing for ARM afaik
< wumpus>
(it might in the future if someone adds support for the ARM crypto intrinsics to bitcoin)
< wumpus>
it'd be nice to be able to run gitian on non-x86 platforms, doing at least one such build would add reassurance against intel ME antics
< sipa>
syncing on my kdroid was painfully slow as it flushed every few blocks in order to prune
< sipa>
by making it only prune whwn kver 2x the prune target i made it an order of magnitude faster, i think
< sipa>
*odroid
< Sentineo>
I run an odroid as well, the arm optimisation did help it a lot
< Sentineo>
but my node is not pruned
< Provoostenator>
sipa: I'll keep my Xiaomi A1 around in case someone want to get some measurements for more efficient pruning during IDB...
< Provoostenator>
It took a little under 2 weeks
< Provoostenator>
Using ABCore; I don't know if that uses any optimization.
< esotericnonsense>
on pruning there's the seemingly longer-term thing of having writeback not flush (i haven't been paying attention to this for a while)
< wumpus>
I don't think they're using our ARM builds, at least they didn't use to
< Provoostenator>
esotericnonsense: 11658 is wonderfully small
< esotericnonsense>
i've only just seen it, i'm ill at the moment and trying to reason through the code isn't working. if it actually does what it says on the tin i like it and agree with promag that 10% is too little.
< esotericnonsense>
wondering if there's some weird interaction with minimum prune size like if you are at 550MiB and use 50% can you end up with very few blocks.
< Provoostenator>
wumpus: I just nagged the author on Twitter, we'll see
< wumpus>
might make sense to make their ndk build part of the standard gitian targets at some point
< Provoostenator>
Maybe it should never go below 550, but any prune size above that will use some intelligent percentage.
< wumpus>
anyhow if they reply please let me know what they use now
< wumpus>
jonasschnelli: Provoostenator: I also wonder if abcore added android ndk to the depends system (but never upstreamed it) or they're building their own dependencies somehow
< sipa>
GAit is the author of abcore
< wumpus>
oh lol
< Provoostenator>
No, we should reach him through a screenshot of Twitter posted on Reddit with a link to this IRC log...
< wumpus>
hahahahah
< sipa>
Provoostenator: excuse me, but i'm sure you meant a screenshot of a slack with a bridge of this IRC channel :p
< Provoostenator>
That too. How hard would it be to make that bridge two-way, somehow mapping and authenticating Slack names to IRC names for those who use both?
< wumpus>
I'm would be surprised if there's not already software for that and you just have to stash it somewhere and configure it. But to be honest I'd prefer no bridges *to* this channel, in the interest of managing signal-to-noise.
< Provoostenator>
I suppose most people who can compile from source can figure out IRC.
< GAit>
lol
< GAit>
so abcore used to use debian/archlinux build until core provided arm binaries. Now it uses ndk native builds, using github.com/greenaddress/bitcoin_ndk for the building part
< GAit>
would love some feedback, it's very hacked up together (though I'm not too worried as abcore is experimental and in alpha)
< GAit>
to enable the arm optimization i will need to enable armv-7a (for Thumb-2)
< GAit>
I'm using clang as android ndk deprecated gcc and they are going to remove it
< GAit>
however i had to use an older ndk than the latest because the latest has some issues/bad headers
< wumpus>
GAit: one suggestion would be to benchmark whether thumb or arm instructions gives the best performance. I vaguely remember that for me, simple ARM32 instructions gave the best performance at some point at least on the secp256k1 benches, on i.mx6.
< Provoostenator>
wumpus: nice. I suppose you don't really need individual user authentication for this, just as long as all messages end up on both services (without feedback loops).
< GAit>
wumpus: you mean things like field_10x26_arm.s
< wumpus>
GAit: I mean the code generation of the compiler, what kind of instructions it generates, whether it uses the processor in ARM or THUMB mode (or interwork)
< GAit>
indeed to make sure one would need to benchmark. but before you can benchmark you need to get it to work, and i will have to enable v7a/thumb-2 otherwise i get rc/asm/field_10x26_arm.s:109:2: error: instruction requires: thumb2
< wumpus>
THUMB usually results in more compact code, but ARM instructions allow more flexibility, so doing more in less instructions. Then again this was relevant for ARM32, probably not for AARCH64.
< wumpus>
GAit: just looked at the bitcoin_ndk build a bit and doesn't seem like it needs huge patches, I guess the introduction of src/ifaddrs.c is the biggest change
< GAit>
wumpus: yes, it doesn't need much patching. Not with this ndk version anyway. Had to do a lot more with later and still didn't manage so i gave up.
< wumpus>
I vaguely remember needing to do the "i686_linux_CC=$(default_host_CC) -m32" change in linux.mk as well. This was for building depends on ARM64, for ARM64. Depends is really rooted in 'we build from x86 linux' right now.
< wumpus>
hmm okay
< GAit>
I'm terrible at make/build otherwise i'd see if i could make a PR or more out of it
< wumpus>
so building bitcoin core on most recent ndk is an open issue?
< GAit>
(assuming there was interest)
< wumpus>
there's certainly interest
< GAit>
yes, i can do that on a branch so that you can see the travis log
< wumpus>
going to open an issue for that
< wumpus>
GAit: just opened #11844 feel free to post that there so people have an idea what the open problems are
< GAit>
great, thanks wumpus . Oh I didn't know about termux having core builds, i searched in their repos. I was going to try that after failing on NDK but eventually trying different things got it to work
< esotericnonsense>
GAit: it doesn't, i built it on termux
< esotericnonsense>
i don't have the time to maintain it so didn't submit to packages
< wumpus>
agree, distro packages for bitcoin are a bad idea if they're not maintained
< wumpus>
tends to keep a share of the users permanently stuck in the past
< Sentineo>
bitcoins build system is awasome anyway, do not feel the lack of an official distro package per distribution
< wumpus>
yes it's pretty neat, quite uncommon for open source projects to ship with scripts to easily build dependencies
< Sentineo>
the only thing I could not manage is the gitian build
< Sentineo>
but I did not try hard enough :)
< GAit>
wumpus: indeed I would have never managed without the build system being 'generic' and being battle tested against arm
< wumpus>
yeah gitian could definitely be easier
< Provoostenator>
I still have some notes for a future Gitian documentation improvement PR, but I'd like to do a few more builds to wrap my head around it.
< Sentineo>
Provoostenator: if you have a draft I would be happe to at least give you feedback :)
< Provoostenator>
I'd like to get it to a point where anyone with OSX (others can do Linux / Windows) and very little sysadmin skills can sign a build.
< Provoostenator>
Sentineo: my scribbled notes are worse than the actual documentation :-)
< Sentineo>
:) ok, just in case ... let me know
< wumpus>
but better than nothing at all probably
< Provoostenator>
I also wonder if it's possible to verify an OSX signed binary without gitian signers having to sign off on the signed binary.
< Provoostenator>
I.e. isn't it possible to unpack a signed binary, strip the signature and then check that against the unsigned gitian build?
< Provoostenator>
I'd need to study how OSX code signing actually works.
< wumpus>
it's possible, but if gitian signers don't sign off on the signed binary that makes verification of hashes a lot harder. No longer can you just use sha256, but need a script, which might have been tampered with itself!
< Provoostenator>
Right, so that would only make sense if the script is trivial enough that it fits on a T-shirt.
< wumpus>
we looked at the inverse approach that you describe back in the day that we introduced signed executables, but decided not to because of drawbacks like that
< wumpus>
right, and it takes some serious binary surgery so that's likely not the case and will have dependencies of itself
< Provoostenator>
"Binary surgery" - lol
< Provoostenator>
It's weird though; these executables work, so you'd think everything is already in the system...
< bitcoin-git>
[bitcoin] laanwj closed pull request #11602: utils: removed deprecated check and function for OpenSSL compatiblity (master...old_openssl_names) https://github.com/bitcoin/bitcoin/pull/11602
< GAit>
wumpus: so for arm/thumb i can't just set global CFLAGS because otherwise native_ccaache will pick them up and break, suggest hacking linux.mk or is there an easy way maybe to disable native_ccache
< wumpus>
could add a NO_CCACHE that works similar to NO_WALLET NO_QT etc, but no currently that doesn't exist
< wumpus>
but I agree that is inconvenient, maybe cfields has a simpler sugggestion
< GAit>
well disabling it would be a hack, if itn's not there i think is better that i patch directly linux.mk to add my cflags there. Or, I could set them maybe just for the core build and not for the dependencies.
< wumpus>
"passing global cflags for the target" would be nice as a supported option for depends, no matter ccache
< GAit>
yes indeed it would
< wumpus>
I've run into similar issues in experimentation but also ended up patching makefiles
< GAit>
i feel terrible/funny because i assumed it was possible and felt too embarassed to ask how so i've been looking and patching when i didn't find a better way
< GAit>
anyway for now i'll try to set the flags just for the core build and not for the deps, unless that's stupid for some reason
< GAit>
by the way, I think it's curious how ccache picks up my cflags but not my cc or cxx env vars
< wumpus>
none of the dependencies that are not built into bitcoin's source tree are performance critical (maybe with the exception of boost, but that's mostly header only?)
< wumpus>
so if you're experimenting with build flags setting them just for the core build is probably fine
< bitcoin-git>
bitcoin/master 529b866 MeshCollider: Test datadir in conf file exists
< bitcoin-git>
bitcoin/master 7630a1f Wladimir J. van der Laan: Merge #11829: Test datadir specified in conf file exists...
< GAit>
wumpus: cool, ta
< wumpus>
(there's a similar consideration on whether to do flto on just bitcoin core or all the dependencies as well. Though that experiment was put on hold when it turned out flto somehow negatively affected performance, at least in a specific case)
< GAit>
wumpus: i've updated the issue you raised with the specific failure, has to do with NDK doing something stupid and the woraround would be to not set FILE_OFFSET_BITS
< GAit>
only breaks on 32 bit, on 64 bit it works
< wumpus>
okay at least easy to work around then, good
< GAit>
well it wasn't that easy for me, i tried to unset it in a couple of places but probably i just misunderstood how it worked
< GAit>
or what was setting it. I think core sets it
< GAit>
but maybe some of the deps too?
< GAit>
anyway after a long intensive and unfruitful discussion with the ndk that's when i just went to an earlier version
< GAit>
yes so i tried to unset it there but i think it still failed, i.e. something else was setting it too, or at least that's what i recall
< wumpus>
indeed,if that was all then we could just stop setting it when ndk build detected
< GAit>
the asm flags passed to core only break the configure, doesn't find boost sleep, i assume both (or neither) need them can't do it half and half
< wumpus>
can you pastebin configure.log part where the boost sleep test fails? it has the detailed error message
< wumpus>
the "cannot find boost sleep" error is completely useless in that regard it can mean so many things
< wumpus>
would make sense to structure the boost detection differently, so that it first tries if it can compile/link anything against boost and only then try to find the sleep function to use
< wumpus>
errors like /build/bitcoin-0.15.1/depends/arm-linux-androideabi/share/../lib/libboost_system-mt.a(error_code.o)(.ARM.extab+0xc): error: undefined reference to '__gxx_personality_v0' look scarier
< wumpus>
looks like it can't even link the c++ standard libary
< wumpus>
so yes, apparently that doesn't work
< GAit>
for what is worth this is master of bitcoin_ndk with extra "-march=armv7-a -mthumb" cflags for the core part (not the deps) of arm (not arm64)
< GAit>
cflags and cxxflags to be precise
< wumpus>
strange
< wumpus>
I wonder which of those two is the culprit
< wumpus>
my guess would be at changing the arch
< wumpus>
there are some compiler settings that effectively need a new build root but I've never noticed thumb/arm to be one of those
< wumpus>
oh I remember it depends on the setting of interwork, ift that's not enabled you can't mix thumb and ARM. But it's pretyt much the default for a long time for linux distros at least.
< GAit>
maybe because i'm using clang the toolchain is not assuming linux? not sure
< wumpus>
it could all be different for ndk which is not really linux
< GAit>
kernel is linux, but no gnuland
< wumpus>
kernel is a specially patched kernel to add bind etc, or are the android patches upstream now?
< wumpus>
binder*
< wumpus>
but yes that part is compatible in one direction, you can run debian rootfs' on android, but not android rootfs on debian
< promag>
wumpus: if you feel like merging something 11838 :P
< bitcoin-git>
[bitcoin] sdaftuar opened pull request #11847: Make boost::multi_index comparators const (master...2017-12-fix-const-comparator) https://github.com/bitcoin/bitcoin/pull/11847
< jonasschnelli>
re BTC message signing: what is the "vchSig[0] = 27 + rec + (fCompressed ? 4 : 0);". Is that packing the recid together with the compress-flag into a single byte?
< jonasschnelli>
Why 27?
< achow101>
meeting?
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu Dec 7 19:00:51 2017 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< * BlueMatt>
is working on reviewing it now...lots of "ehh, you should clean this up instead of hacking around it" type things which may get pushed to a new pr, but nothing broken yet
< sipa>
hi, only half here
< BlueMatt>
re: high prio #11383 should probably be taken off
< instagibbs>
will retest segwit wallet/qt pr on top of #11839
< jonasschnelli>
I'll try to take over 11383 (we should have this in 0.16)
< gribble>
https://github.com/bitcoin/bitcoin/issues/11839 | dont attempt mempool entry for wallet transactions on startup if alr… by instagibbs · Pull Request #11839 · bitcoin/bitcoin · GitHub
< promag>
hi
< BlueMatt>
oh, yea, 11839 should be tagged 0.16
< BlueMatt>
its a trivial fix, I think
< BlueMatt>
jonasschnelli: no need to take it over if instagibbs is active on it?
< wumpus>
if it's a trival fix it should probably be merged instead of tagged?
< instagibbs>
BlueMatt, wrong PR, just looks like he mentioned it i think?
< jonasschnelli>
I'm doing the Multiwalltet GUI PR
< instagibbs>
that's multiwallet
< BlueMatt>
ohoh, yea, wrong #
< * BlueMatt>
suggested an alternative fix for 11839, so thats just pending disucssion of "the right fix"
< BlueMatt>
all options are relatively trivial
< BlueMatt>
#11824 may be mergeable with another review to fix reindex on master, jamesob indicated his issue was the result of "running wrong build"
< achow101>
I did notice another OOM issue I think, but it looked like to be hard to reproduce and required significant uptime
< BlueMatt>
achow101: I failed to find any hidden memory in massif, but I'm still doing runs in it so will look further
< wumpus>
do we have a memory leak?
< BlueMatt>
I dont (currently) believe so...master will OOM during reindex cause validation runs ahead of validationinterface and memory grows huge, but its not technically a leak cause it will catch up eventually if you have enough memory to do it
< wumpus>
I haven't had any OOM crashes FWIW
< BlueMatt>
11824 fixes that
< jonasschnelli>
Did also a quick run with my leak tool and some stuff popped up. leveldb, bdb ansd some other things
< jonasschnelli>
Will have a closer look soon
< wumpus>
oh, only during reindex, I haven't done that recently
< BlueMatt>
topics?
< BlueMatt>
wumpus: could also happen directly after restart if you've been offline for a month or whatever
< achow101>
or during IBD
< wumpus>
I think an OOM issue is a pretty serious topic? but other suggestions are welcome of course
< morcos>
if anyone objects to my "won't fix" of #11800, speak up now. will update code comments per ryanofsky suggestion
< gribble>
https://github.com/bitcoin/bitcoin/issues/11800 | Bitcoin is returning higher fees for 36 block window than 2 block window (on testnet) · Issue #11800 · bitcoin/bitcoin · GitHub
< BlueMatt>
well I think we've at least gotten 97% of it down, several folks looking into the last 3 and it may be a false positive
< Provoostenator>
OOM?
< michagogo>
Out Of Memory
< achow101>
wumpus: the only known OOM right now is fixed by 11824
< BlueMatt>
achow101: ibd seems less likely cause time spent downloading blocks is time for wallet to catch up...
< wumpus>
achow101: ok
< wumpus>
on to morcos' topic
< * BlueMatt>
makes a quick note that, of the high-priority-to-review items, #11363 is very easy to review and has been sitting for a while
< wumpus>
#topic Bitcoin is returning higher fees for 36 block window than 2 block window (on testnet)
< BlueMatt>
jonasschnelli: morcos had a writeup on the issue
< jonasschnelli>
#?
< cfields>
BlueMatt: thanks :)
< BlueMatt>
#11800
< morcos>
11800
< gribble>
https://github.com/bitcoin/bitcoin/issues/11800 | Bitcoin is returning higher fees for 36 block window than 2 block window (on testnet) · Issue #11800 · bitcoin/bitcoin · GitHub
< wumpus>
morcos posted that
< wumpus>
right
< wumpus>
so the situation is pretty much unique to testnet and extremely unlikley on mainnet
< jonasschnelli>
IMO low prio or even "won't fix"
< BlueMatt>
yea, I mean I think I'd prefer to not call it "wont fix", but certainly not anything worth spending time on compared to other priorities
< wumpus>
I tend to agree, very low priority if it only affects testnet's wildness
< jonasschnelli>
what BlueMatt said
< wumpus>
we kind of know that the current testnet isn't realistic
< wumpus>
a frequent request is a more realistic testnet FWIW
< BlueMatt>
suggested topic: testnet4
< midnightmagic>
the data in tn3 is fairly valuable on its own. if any tn reset is being considered for a pullreq it would be really nice to effect a high-quality preservation of tn3. like an option or something to put it into archive-mode.
< BlueMatt>
you mean as test-cases?
< promag>
next topic? btw I would like to have more NACK/ACK on #11826
< gribble>
https://github.com/bitcoin/bitcoin/issues/10267 | New -includeconf argument for including external configuration files by kallewoof · Pull Request #10267 · bitcoin/bitcoin · GitHub
< wumpus>
#topic config file handling
< meshcollider>
We have to work out what configurations we should support, e.g. whether -conf should be repeatable, whether we have -includeconf, -netconf and -conf, etc.
< wumpus>
so the question is pretty much whether to do per-network config file
< jonasschnelli>
IMO the config layers are already complex... not sure if we want to add more
< meshcollider>
its getting a bit messy
< wumpus>
or -testnet-X and -regtest-X
< jonasschnelli>
There are serval levels and multiple files
< wumpus>
where X are options like port, walletdir, logfile, which are only useful per network
< wumpus>
currently if you define e.g. port or bind in bitcoin.conf you will get collisions when you run both testnet and mainnet on the same machine
< meshcollider>
one idea I had was suggested in https://github.com/bitcoin/bitcoin/pull/10996#issuecomment-346189099, basically we default to using root-level bitcoin.conf and network specific network.conf if they exist, but if -conf is specified then we just use that and not the network specific one too
< wumpus>
so one potential solution for that was to do per-network config files, but as said that makes things reallly complex
< meshcollider>
and then allow -conf to be repeatable?
< wumpus>
so an alternative proposal was just to do -regtest-port -testnet-port etc
< jonasschnelli>
wumpus: +1
< aj>
from 10267, having conf repeatable or includeconf recursive seems hard to implement well; currently it only handles a single additional config file aiui
< wumpus>
and I think I like that
< meshcollider>
wumpus what about unique addnode's for each network
< meshcollider>
so just allow -regtest or -testnet to be prefixed to any existing arg?
< promag>
yes, I think that is pretty simple and clear
< promag>
if not set, defaults to?
< wumpus>
these can be specified in any config file or on the command line, no difference, no *context sensitivity* like per-network config files
< jonasschnelli>
promag: the default :)
< wumpus>
yes,the default
< meshcollider>
and then still support #10267 ?
< wumpus>
meshcollider: yes
< gribble>
https://github.com/bitcoin/bitcoin/issues/10267 | New -includeconf argument for including external configuration files by kallewoof · Pull Request #10267 · bitcoin/bitcoin · GitHub
< wumpus>
includeconf is orthogonal
< wumpus>
I see no reason why not
< meshcollider>
yeah but it would allow -regtest-includeconf=whatever
< promag>
jonasschnelli: default of the network right?
< meshcollider>
so it could be good :)
< jonasschnelli>
promag: yes
< wumpus>
it's pretty standard these days to allow including config files from config files for daemons
< morcos>
wait, just so i understand, if you run ./bitcoind -regtest , then do you also have to add -regtest- to each of your command line options
< morcos>
that would be super annoying
< cfields>
an alternative to that would be sections in a config file. and on the cmdline they'd look like namespaces. so, [testnet] port=5. or -testnet::port=5.
< wumpus>
almost any program supporting config files supports that in any way
< promag>
wumpus: yes, like inherit other config
< morcos>
cfields: that soudns way better
< wumpus>
cfields: fine with me too
< wumpus>
it's just that per-network config files make things too complex, so I'd like to avoid that
< meshcollider>
morcos: I guess anything without a prefix would be used no matter what network was chosen
< wumpus>
which kind of namespacing whether it's [net]option or -net-option I don't mind much
< cfields>
and in time the sections could potentially be broken out an included/inerited. but the simple change seems like a simple enough first step to me.
< wumpus>
meshcollider: yeah...
< jonasschnelli>
but morcos point is valid. What if you just start use CLI params and switch between networks...
< cfields>
simple == simple. nice.
< morcos>
seems like it would make a lot of sense to have a single conf file, that has sections : [default] [mainnet] [testnet] [regtest] and you always end up reading default and one of the others
< jonasschnelli>
do you need to add/switch the namespace all the times?
< jonasschnelli>
meshcollider: solves with the namespaces described by morcos.
< jonasschnelli>
*solved
< meshcollider>
promag: I like 11826
< wumpus>
#topic activity feature
< meshcollider>
concept ACK from me
< jnewbery>
seems like the namespace model gives you separate conf files for free: just add a -includeconf in your relevant network namespace
< jonasschnelli>
Yes. We should have that
< wumpus>
I think everyone likes 11826
< jonasschnelli>
Bitcoin Core does a lot of things under the hood... and there is no way to see/control that
< promag>
I have one problem there
< * BlueMatt>
isnt sure about the use-cases for 11826
< jonasschnelli>
An activity window is something should have IMO
< promag>
should the activity have a boost::variant<> source?
< BlueMatt>
seems like putting the cart before the horse, as it were
< BlueMatt>
like, the only things that can go *into* the activity window are things you do from rpc debug window
< promag>
or should we have class WalletActivity : Activity ?
< jonasschnelli>
BlueMatt: most stuff in there should be read-only...
< promag>
BlueMatt: why?
< sipa>
promag: seems like an implementation details
< BlueMatt>
in that case, building an activity window should probably come after making rescan, etc things that users have access to
< sipa>
BlueMatt: reindex-chainstate too
< BlueMatt>
isnt reindex-chainstate handled like ibd?
< BlueMatt>
(as it should be)/
< sipa>
yes, it is
< BlueMatt>
(cause we cant get a progress indicator for reindex-chainstate)
< sipa>
?
< BlueMatt>
I mean the only progress indicator you can do is the same as ibd
< promag>
BlueMatt: in those cases it's an indeterminate progress bar
< jonasschnelli>
Maybe we should first fix the rescan GUI "abortness", then continue with the activity window (will take a while)
< jonasschnelli>
A single progress bar does certainly limit use cases.
< wumpus>
yes
< * BlueMatt>
is just saying, if someone wants to build an activity window, ok, have fun, but I'm not sure where it actually makes all that much sense, cause we dont have many activities that users almost ever do)
< jonasschnelli>
BlueMatt: not only the user triggered ones...
< promag>
BlueMatt: also, activities could stays on the window even after finished, like a log, or your download history
< BlueMatt>
download history?
< sipa>
i do agree there are relatively few use cases now, but the concept is useful... and i think eventually we want to be able to do things more concurrently anyway
< BlueMatt>
you mean rescan history
< promag>
no, I mean browser download history
< jonasschnelli>
Yeah.. a mix between Activity and short term log would be possible
< jonasschnelli>
but hard to make it right
< promag>
it's more friendly than a raw log
< phantomcircuit>
BlueMatt, lots of users complain about ibd apparently stalling, something that indicated what was happening would be good
< jonasschnelli>
[disappearing message] new block validated \n [disappearing message] new peer 1.1.1.1 connected
< BlueMatt>
I mean if I'm a user, I dont really want to see a progress history thing, tbh....either I have all the transactions in my wallet or I dont...
< wumpus>
for that, a debug log view would be more useful
< wumpus>
let's not conflate ongoing activities with a log
< cfields>
:q
< cfields>
er, heh
< jonasschnelli>
heh... Yes.
< jonasschnelli>
previous attempts that are more or less a log where kinda accepted #5896
< BlueMatt>
but, ongoing activities are like....rescan, just rescan, and thats not something you should do all that often...ibd/reindex/sync is a rather separate thing
< BlueMatt>
but, anyway, it sounds like I'm the only one who disagrees, so I'm happy to shut up :p
< wumpus>
sending a transaction is also an ongoing activity
< wumpus>
ideally the GUI would move to do all those things asynchronously instead of in the GUI thread
< BlueMatt>
I mean it sounds like y'all want to restructure our entire gui around an activity log...
< achow101>
I also don't really see the utility of an activity window. but I don't really care either way
< promag>
un/loading wallets too
< wumpus>
no, I want to structure the GUI asynchronously
< wumpus>
an activity thing would be a nice thing that comes with it
< BlueMatt>
sending txn is rather async already, no? I mean you send and then you wait on confirms, maybe with a feebump
< BlueMatt>
(at a user level, that is)
< jonasschnelli>
Lets promag work on that concept and see where it leads to. It's certenly good to have it around in case we can do more stuff in parallel and have other cases where we want to show progress
< wumpus>
the point is that comments are executed in the GUI thread, with the various locks held
< wumpus>
commands*
< jonasschnelli>
Also it would be useful for IPC (GUI / node detatch)
< instagibbs>
probably makes more sense if the wallet is more goal-driven, for now I don't mind the feature
< instagibbs>
(re: wallet comments)
< wumpus>
which is not a nice user experience
< promag>
ok guys, I'll file the PR soon
< wumpus>
e.g. what I've noticed is that when it's catching up to the chain, getting the information for coin control hangs the entire GUI for half a minute sometimes
< wumpus>
which would be fine if it showed a progress indicator but not if everything just blocks
< Randolf>
That would likely cause many users to think it crashed.
< wumpus>
well the window manager starts to think that too
< wumpus>
and wants to kill it
< jonasschnelli>
The GUI synchronousness does sometimes lead to terrible UX.
< Randolf>
Yes. And under the Windows OS, the user sees a message along the lines of "Task not responding" with an option to kill the task.
< wumpus>
jonasschnelli: it was always my intent to change that but I just don't get around to it
< jonasschnelli>
wumpus: it's also pretty complex
< wumpus>
nah, not really complex, just work
< BlueMatt>
yea, cs_main-y-ness of gui sucks atm :(
< Randolf>
So, if the GUI is on a separate thread, then it can always respond to the OS and the user's operations (e.g., open the Help menu, load/save wallets, print reports, etc.) while waiting for updates from the other threads.
< jonasschnelli>
Randolf: we are trying that (ask ryanofsky) :)
< wumpus>
Randolf: right
< Randolf>
jonasschnelli: Cool!
< BlueMatt>
anyway, more topics in 10 minutes?
< sipa>
i have one
< sipa>
would a libgmp dependency by acceptable?
< BlueMatt>
for....secp? or your muhash stuff?
< Randolf>
jonasschnelli: I've done a lot of Java development, and this is how JFC/Swing and the newer JavaFX handle things. The use of atomic variables and thread-safe queues becomes pretty important. Maybe the original GUI for Bitcoin was designed with more development convenience in mind just so that the
< Randolf>
project could get completed faster? I don't consider this to be a bad thing, necessarily, but I'm glad to know that there's an interest in improving this.
< BlueMatt>
I mean doesnt secp optionally use it already?
< wumpus>
#topic libgmp dependency
< wumpus>
libgmp is GPL right?
< sipa>
BlueMatt: it does, and there is a small performanc benefit from using it for validation itself
< wumpus>
if it's MIT/BSD licensed it's ok with me
< meshcollider>
wumpus: yep I think so
< wumpus>
but if it's GPL, we have to say no
< sipa>
but for UTXO hashes it would be a huge difference
< sipa>
i see
< cfields>
sipa: i assume you mean a non-optional dep?
< sipa>
10434 uses the modular multiplication group approach, which doesn't need this - it's faster but much harder to cache
< sipa>
the alternative approach is using EC based rolling hashes, for which using jacobi symbols would be a 2x speedup or so
< sipa>
wumpus: so a 3rd option is not implementing EC-based rolling hashes
< promag>
out-of-battery o/
< wumpus>
also after all the work that was done to lose dependency on openssl for bignums, to introduce dependency on a new arbitrary precision library seems also a reversal to me
< sipa>
so i just wanted to bring it up to see what the option were
< BlueMatt>
so if we use lgpl gmp as a dep, and then someone wants to ship a bitcoincore-modified binary with all deps static-linked, they'd be screwed?
< sipa>
wumpus: to be clear, nothing would be consensus critical (even if rolling hashes were somehow made a consensus rule)
< aj>
BlueMatt: lgpl means they'd have to release sources or their .o/.a files
< BlueMatt>
yea, ok, i meannnn, probably fine, but that is a real change in effective license of bitcoin core
< morcos>
sipa: 2x speedup in what exactly
< morcos>
when would those computations happen and what is the base latency
< cfields>
sipa: at the risk of going too far off-topic, aren't there curves with potentially quicker and guaranteed O(1) map operations?
< morcos>
piont being, perhaps we can punt on the question of libgmp until it becomes clear that optimizing the speed is an important tradeoff to make
< wumpus>
good point morcos
< instagibbs>
morcos, +1
< sipa>
cfields: let's discuss outside of the meeting
< sipa>
morcos: updating a rolling hash given a set of UTXOs created and spent
< cfields>
ok
< BlueMatt>
well meeting over anyway
< gmaxwell>
ugh was missing meeting.
< instagibbs>
gmaxwell, 2 minutes
< instagibbs>
1 minute
< morcos>
right, so how often are we envisioning doing that, does it happen async to everything else, and how long does it take without libgmp
< gmaxwell>
why did we start talking about libgmp? :(
< sipa>
in an absolutely first implementation, i would expect that that would just affect gettxoutsetinfo or its equivalent that just computes the hash from the UTXO set from scratch
< BlueMatt>
gmaxwell: blame sipa
< gmaxwell>
sipa: we do not want gmp as part of bitcoin's consensus critical paths, totally independently of using it as a dependency.
< sipa>
$ git blame sipa
< sipa>
fatal: cannot stat path 'sipa': No such file or directory
< BlueMatt>
yes, that point was made
< sipa>
gmaxwell: absolutely
< sipa>
no intention of changing that
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Dec 7 20:00:26 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< gmaxwell>
ah, you were talking about it for the jacobi symbol for aggregate validation?
< aj>
libzmq is lgpl isn't it?
< wumpus>
yes
< wumpus>
aj: no
< sipa>
more in the context of rolling hashes
< wumpus>
aj: oh, you're actually right
< wumpus>
aj: wtf, I never knew
< wumpus>
aj: however zmq is optional
< BlueMatt>
that sucks...our release binaries have lgpl crap in them and we dont have an easy way to get a bitcoind that is linked to everything but zmq?
< aj>
the release binaries have zmq now?
< * BlueMatt>
wonders if some altcoins are violating that license
< BlueMatt>
almost certainly are
< achow101>
releases have zmq and have had them for a while now
< wumpus>
--disable-zmq
< wumpus>
(or was it --without-zmq, I don't remember)
< wumpus>
my preferred way for not linking against something is just to not have it installed on the VM
< jonasschnelli>
aj: What I really would like to have is something like this in a web-base script (CGI)
< sc__>
does anybody know by chance, why bitcoind regtest is missing witness data in p2p messages?
< sc__>
and how to get witness data via p2p?
< instagibbs>
sc__, make sure you actually activated segwit by making a few hundred blocks...
< jonasschnelli>
aj: That web-site/app could be an interactive mode of that script you wrote with some cookie based local editing... one could flag the PRs he wants to review (locally stored)
< wumpus>
for me a console-based tool is better
< jonasschnelli>
wumpus: nerd! :)
< sc__>
instagibbs: I really did this (the block count is over 11000 now)
< jonasschnelli>
wumpus: Yeah.. consol based would even make more sense...
< wumpus>
jonasschnelli: just extremely paranoid :)
< jonasschnelli>
not sure if ncurses is more complex then HTML
< jonasschnelli>
wumpus: But I guess at some point, you have to link to the github issue to read more about it... or do you use lynx or some sort of console browsing?
< instagibbs>
aj, very cool
< wumpus>
jonasschnelli: sometimes, but lynx doesn't work very well/at all on github
< instagibbs>
aj, maybe the submitter shouldn't be able to advance their own PR's labels, heh
< jonasschnelli>
aj's tool should also include the discussion,.. ideally it would be a full fletched github console client... :)
< * jonasschnelli>
stops dreaming now
< wumpus>
jonasschnelli: I think it's a good argument for the tool to be low-level, just print the data in some format - anything like a UI can be added on top
< aj>
instagibbs: oh, did someone ack their own PR? i didn't put a check in for that in the python version
< jonasschnelli>
sure. However, well done aj
< instagibbs>
aj you did in your PR testing it :P, unless I missed something
< jonasschnelli>
kallewoof: re your BIP174,... I guess this also works perfectly fine for pure unsigned transaction...
< aj>
instagibbs: oh, yeah, in that i specifically disabled the check so i could test it :)
< wumpus>
and aj wrote it in python instead of (s)hellscript, I like that
< aj>
wumpus: oh, i have some curl|jshon for loops i was using earlier if you prefer :)
< jonasschnelli>
kallewoof: the "partially signed" tag makes it look that is "partially" unusable for pure softwallet<->hardware wallet interaction
< instagibbs>
aw darn :) thinking of writing that interface anyways...
< achow101>
jonasschnelli: hmm?
< jonasschnelli>
achow101: re your BIP174...
< instagibbs>
jonasschnelli, yes the idea is that createraw, or something, would create a thing that is potentially signable
< jonasschnelli>
I'm looking for a standard how software watch-only wallets (assume Core) can talk to a hardware wallet
< jonasschnelli>
Need to read more into the BIP... but seems 174 does cover that case pretty well
< wumpus>
seems there is suddenly lots of interest in signing messages
< aj>
jonasschnelli: anyhoo, i could turn that RESULTS.txt into some html with links and nicer formatting, updated by cron once an hour or something without too much trouble; making it properly dynamic/current would take some effort though. worthwhile?
< achow101>
jonasschnelli: how would it be unusable?
< jonasschnelli>
wumpus: yeah! I'm feeling like I'm missing a party... don't know what people do sign now all the times
< instagibbs>
wumpus, this time it's transactions, we swear!
< jonasschnelli>
achow101: Assume I have a Core in watch only mode and did a fundrawtransaction with watch only inputs... then I'd like to hand "it" over to the hardware wallet.
< jonasschnelli>
achow101: What data and what schemantics do I use
< jonasschnelli>
BIP174 would be a way? woudln't it?
< achow101>
yes, bip174 would
< jonasschnelli>
because there is no other standard how to serialize a data-blob including the inputs and other elements required for signing
< instagibbs>
that was the reason for the bip, ye
< achow101>
the hardware wallet interaction was one of the motivators for making the bip
< jonasschnelli>
achow101: would a mime type make sense?
< jonasschnelli>
instagibbs: or what channels to transmit your data-package have you considered?
< jonasschnelli>
achow101: yes... hard to figure out what ways could be standartized
< wumpus>
aj: oh wowthis is pretty neat, even allows setting permissions per token
< aj>
wumpus: yeah, doesn't suck.
< wumpus>
I guess it needs no permissions at all as it's read-only?
< aj>
wumpus: yup
< wumpus>
I wish github ssh keys could be configured in the same way, e.g. a certain key can be set to only allow to push to a certain project
< aj>
wumpus: guess they want you do do that by protecting branchs and pull request statuses (like "can't merge if travis doesn't give the ok")
< wumpus>
aj: yes that can only allow certain users to push to certain branches, which is extremely useful
< wumpus>
aj: but in my case it'd be useful to have a certain VM/user only have keys to push to a certain repo and not all repos my account has access to. I could create multiple accounts but meh.
< wumpus>
aj: anyhow github has improved a lot with regard to more granular access control in the last few years which is very good
< aj>
wumpus: i think you could make a "github app" which you could then assign to a repo and which could maybe then push to non-protected branches...
< wumpus>
aj: yep non-protected branches would be fine; allow it to push to laanwj/bitcoin but not bitcoin/bitcoin master for example
< aj>
wumpus: hmm, i might see if that works actually. i was thinking about having lightningbot update a repo of its meeting logs automatically (and use an Attendees file there to pick people to highlight), but got stuck on how to push... https://github.com/ajtowns/bitcoin-core-meetings
< wumpus>
aj: I've enabled the token and with that, check_acks ran to completion without error, thanks
< promag>
wumpus: considering the current lockunspent help description, #5584 can be closed right?
< wumpus>
promag: I'm not sure; what does the help description change about a feature request?
< promag>
ah right, feature request
< promag>
not really something important to implement right?
< wumpus>
no
< wumpus>
well there are certainly use-cases for persistently locking UTXOs in the wallet
< wumpus>
less if 'abandontransaction' always worked
< wumpus>
the most common use case of locking UTXOs persistently would be a transaction, I think where the use case for explicit locking comes from is to be able to unlock them easily again, which isn't possible with the current wallet code
< wumpus>
how long does it take for github to forgive you for rate limit exceeding? I can't use github-merge.py either anymore from this IP :-)
< promag>
wumpus: I know a use case where there is explicit locking, and because of the lack of lock persistence, the locks are saved outside and verified/repeated