< fanquake>
wumpus I'll have a look into adding configure checks for 4.8+
< wumpus>
fanquake: thanks! I looked around a bit and checking the type and version of the c++ compiler is harder than one'd expect, it's pretty much entirely unsupported
< wumpus>
fanquake: might be most straightforward to just add a "check you compiler version, only gcc 4.8 and higher is supported" message when AX_CXX_COMPILE_STDCXX fails
< wumpus>
(and also add the minimum clang version then)
< wumpus>
as that macro will already fail on gcc 4.7 apparently
< wumpus>
and pass on 4.8
< fanquake>
wumpus yep, I was having a bit of a look as well. Didn't seem that straight forward. Especially since Clang also defines __GNUC__ etc
< fanquake>
No double cfields would have some black magic to get it done..
< fanquake>
wumpus I think you suggestion is good though. I'll put together a PR.
< wumpus>
yeah... what they suggest you really want to check is certain features, not the version number of the compiler, because different compilers have different version number schemes. But checking every single c++11 feature is going to be tiring
< wumpus>
what we realy want to check is "c++11 support at the level of gcc 4.8"
< fanquake>
wumpus Pretty certain GCC 4.8.1 is feature complete for c++11. So I think warning if AX_CXX_COMPILE_STDCXX fails should be enough?
< wumpus>
should we cancel today's meeting because of thanksgiving?
< luke-jr>
I won't be able to make it, at leat
< luke-jr>
least*
< wumpus>
that's probably true for most people from the US
< instagibbs>
still may be a critical mass, though I will not be around
< cluelessperson>
wumpus: if there is a meeting, may I be present?
< wumpus>
cluelessperson: there's no need to ask that
< wumpus>
the meeting is here every week 19:00 UTC, everyone is welcome
< cluelessperson>
sweet, sorry, I've been hanging in channel, but I've been avoiding speaking here as I feel I lack skill sets to be of much help
< wumpus>
every week thursday*
< luke-jr>
if you don't have something helpful to say, don't say it; if you do, say it XD
< cluelessperson>
luke-jr: but my name is accurate!
< mlz>
no meeting today though?
< sipa>
i may be able to attend
< mlz>
Happy Thanksgiving to all Core devs! Thank you for your hard work and dedication! :)
< Eliel>
how much memory does the UTXO set require per txout currently and what's the theoretical minimum it can be pushed down to?
< Eliel>
according to statoshi info, the current serialized UTXO set has ~55500000 transactions and that takes 2.9GB of space. So, that comes out to something like 53 bytes per utxo
< Eliel>
I suppose I'll go with that.
< wumpus>
mlz: thanks!
< wumpus>
Eliel: it also needs to be in an efficient format to make updates, otherwise you end up rewriting the whole thing for every block
< Eliel>
wumpus: would that be more than 53 bytes per txout?
< wumpus>
I mean you can probably save some space by just concatenating the whole thing then putting it through a compressor with a large block size, but except as a snapshot it'd not be useful
< wumpus>
I don't know.
< wumpus>
it will still be of the same order anyhow
< Eliel>
I'm trying to estimate the amount amount of RAM required on full nodes per user for LN users and non-LN users, so I guess the accuracy of the number is not a big issue. As long as it's not off by an order of magnitude :)
< jonasschnelli>
Oh. It's thanks giving... I would be around for the meeting.
< wxss_>
clear
< meshcollider>
I'm flying to Sydney in half an hour or so so I'll only be here for the first part of the meeting if it's on
< achow101>
Meeting?
< Provoostenator>
Suggested topic: onboarding
< sipa>
present
< jonasschnelli>
here
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu Nov 23 19:02:43 2017 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< achow101>
we could still talk about things though
< jonasschnelli>
Sure.
< promag>
topic suggestion, network conf etc
< jonasschnelli>
Provoostenator: what would be the benefit behind that (first PR) project ?
< Provoostenator>
New PR's always get a lot of attention, but after a while they lose attention. That's fine for experienced devs, but I think it discourages new contribuors.
< * BlueMatt>
-> vacation, see y'all next week
< promag>
o/
< wumpus>
Provoostenator: I think it's okay to go into this room and shout at people to review your PR
< jonasschnelli>
Provoostenator: Yes. One needs endurance. :)
< Provoostenator>
Once they get their first PR in, they're probably much more likely to keep contributing and be more assertive / patient.
< jonasschnelli>
If it looses attention, then it's probably not priority or the dev did not shout loud enough
< jonasschnelli>
The main bottleneck is still serious reviews
< kanzure>
hi.
< achow101>
jonasschnelli: sometimes people don't know that they should shout
< wumpus>
I mean it's sometimes sad how PRs go ignored, but it's kind of how open source works, you need to bring attention to your PRs somehow
< kanzure>
i missed the meeting :(
< Provoostenator>
Yes, that's why this Project would only apply to the first PR. After that they can be encouraged to shout more.
< sipa>
or people aren't aware that some things take a long time - and this doesn't mean they're not going to happen
< wumpus>
e.g. if it fixes a issue, then reply to people posting issues to test your patch
< achow101>
kanzure: we decided during the meeting that we wouldn't have a meeting :)
< jonasschnelli>
sipa: indeed
< jonasschnelli>
I think we should maybe add a beginners guide...
< jonasschnelli>
Some people expect merges in 1-2 days.
< wumpus>
add it to CONTRIBUTING.md
< jonasschnelli>
yeah
< Provoostenator>
Yeah, I've seen PR's by very experienced people open for over a year, so indeed peoples should have realistic expectations.
< jonasschnelli>
Adding new PRs to the project would be another thing maintainers have to care about
< promag>
there is also the case that a review sits there for a long time
< wumpus>
bitcoin isn't really a good project for first time open source contributors in that regard, some projects just merge everything effectively instantly, but we cannot have a policy like that, not for first contributors ither
< jonasschnelli>
Provoostenator: recently a 2yr old PR of mine got merged...
< sipa>
my first PR took half a year, and that was in 2011 :)
< wumpus>
promag: oh yes, indeed, some PRs get lots of review then the author pretty much just ignores it
< wumpus>
promag: and they get closed after half a year or longer
< Provoostenator>
@wumpus good point that it's not a good first open source project. Although the quality of reviews is really great for learning.
< jonasschnelli>
Provoostenator: I think you could add a part in CONTRIBUTING.md
< jonasschnelli>
that would be helpful for new contributors
< promag>
right, it's also a bit weird to submit a PR, have reviews, and then it's ignored by the author
< BlueMatt>
CBlockStore is still WIP from 2012 :p
< wumpus>
authors can also get more attention by helping review other patches
< Provoostenator>
I think contributing.md already says it takes a long time and you shoudl go on IRC.
< promag>
true, some authors ignore other PR's, but that's more acceptable IMO
< wumpus>
in any case of a PR really fixes a bug or issue, it won't usually take that long before it's merged unless something is wrong and not being fixed
< achow101>
unfortunately people don't really read the documentation that explains what they should do
< Provoostenator>
Yeah, so maybe something we could add to that doc is to encourage people to find a painful issue.
< sipa>
jonasschnelli: i'm very happy to talk about that
< sipa>
jonasschnelli: i really don't think so
< sipa>
i've considered making a second wallet too, but it
< sipa>
's a pointless exercise i think
< wumpus>
that was discussed so many times over the years
< achow101>
jonasschnelli: I think it would be better to just make a new wallet format entirely and make it completely backwards incompatible
< Provoostenator>
@wumpus new tickets always get tons of attention. It's the stale ones that worry me.
< sipa>
achow101: indeed
< sipa>
achow101: seen my writeup? :)
< achow101>
sipa: yeah
< Provoostenator>
And a new contributor might just pick the wrong topic (like making RBF a default :-)
< promag>
sipa: > pointless exercise i think - why?
< wumpus>
Provoostenator: I don't think it's about newness in this case - the person explained clearly what the issue was, then fixed it, with a straightforward patch
< sipa>
promag: those who don't know history are doomed to repeat it
< jonasschnelli>
achow101, sipa: but wouldn't this end up in have a large amount of code handling the back. compatibilizt?
< wumpus>
Provoostenator: it's also about communication and doing something people care about :)
< promag>
sipa: and those that know history?
< jonasschnelli>
I don't mean rewrite the wallet, I mean copy the wallet souces, remove accounts, remove pools, remove all the upgrade migrations, add new SW stuff
< jonasschnelli>
same same but different
< sipa>
promag: will have much more impact working on existing code, rather than starting over and hoping it will attract review attention
< wumpus>
yeah...
< jonasschnelli>
that's a point
< wumpus>
jonasschnelli: well accounts should be removed out from the current source, not a copy
< sipa>
ack
< wumpus>
jonasschnelli: same for some of those other things
< jonasschnelli>
I just fear the migration at statup thing...
< jonasschnelli>
also,... that we keep BDB4.8 until core 0.25
< sipa>
heh, swapping out the storage format seems orthogonal
< wumpus>
changing the storage format to another database is pretty easy
< achow101>
jonasschnelli: does it need to migrate at startup?
< wumpus>
I changed it locally to leveldb a while ago
< sipa>
jonasschnelli: for the storage format, it think it should be done independently from everything else
< sipa>
so that it is a straight translation from one db to another, and none of the key/values inside change meaning
< sipa>
which means upgrade and downgrade are trivial
< wumpus>
(because I didn't want to port berkeleydb to that environment)
< wumpus>
exactly sipa
< jonasschnelli>
Yes. Right
< sipa>
_independently_ we should think about a new semantic layer (see my writeup, for part of that), which will be an incompatible upgrade at some point i expect
< sipa>
but it doesn't need to happen at the same time as the storage layer change
< jonasschnelli>
sipa: you mean the record type schemantics?
< sipa>
yes
< achow101>
sipa: it seems like a storage layer change would be the easiest way to guarantee incompatibility
< wumpus>
my biggest annoyance about the current wallet is that it reads everything into memory, it's a database ffs
< sipa>
achow101: version numbers work pretty well :)
< jonasschnelli>
sipa: you wrote "Conversion of old wallet to new ones will probably be the trickiest part. It will involve a one-time operation at startup"....
< sipa>
jonasschnelli: yes
< achow101>
sipa: but then you have two incompatible upgrades, versus one
< wumpus>
there's no need to have all transactions and crap going back years in memory
< sipa>
achow101: storage layer wouldn't be incompatible
< achow101>
why would they be compatible? Older software wouldn't be able to read a new storage format
< jonasschnelli>
I gust questioning the endless backward compatibility. If we don't do us a favor and set a point (version X) where the wallet crated with version X will no longer be backware comp.
< sipa>
sure, but both upgrade and downgrade would be trivial
< sipa>
both things can happen in the same release, and that would certainly be more convenient
< wumpus>
jonasschnelli: backwards compatibility is extremely important, though it'd be fine with me if that's a one-time upgrade at some point
< achow101>
right, but then you need something that can downgrade it. if you just downgrade the software, it would be incompatible
< sipa>
but i don't think discussions about changing the storage format should get in the way of semantic changes
< wumpus>
jonasschnelli: but people with old wallets shouldn't be stuck!
< sipa>
and the other way around
< wumpus>
but downgrading seems completely unimportant to me
< jonasschnelli>
wumpus: Yes. This is why I though keeping the legcy stuff but not mixing the code.
< achow101>
ooh we could make CWallet, CDB, CWalletDB, etc. actually make sense then!
< sipa>
hehe
< Provoostenator>
There's also the possibility of importing old wallet from backups rather than old database files. Obviously not a good experience at all.
< wumpus>
achow101: some of the classes need renaming, that's orthogonal :)
< jonasschnelli>
achow101: and there are still some layer violations...
< sipa>
Provoostenator: ?
< sipa>
backups are database files
< Provoostenator>
Oh, it's not using the dump format?
< sipa>
the dump format is just for keys
< wumpus>
no, not if you use walletbackup
< achow101>
Provoostenator: no, it just copies the wallet.dat file to somewhere lese
< wumpus>
dumpwallet/importwallet is separate
< Provoostenator>
I see. Having a backup format that's not a database file would be useful then?
< sipa>
it's complicated
< wumpus>
what do you want to backup?
< sipa>
we have two axes really... secret or not, and mutable or not
< wumpus>
if it's just the keys, dumpwallet is what you want
< Provoostenator>
Keys and metadata.
< wumpus>
if you also want transactions and transaction metadata it's kind of difficult
< sipa>
for example address labels really require a dump after every new address created
< sipa>
stored transactions (especially unconfirmed ones) need a dump after every transaction
< * jonasschnelli>
vanity generated lables!
< sipa>
but with HD wallets, you don't really need backups at all to prevent monetary loss
< Provoostenator>
I guess I'd want two backups: 1) the HD seed, done once 2) everything else, done every now and then
< sipa>
and which of those is more important depends on the use case
< sipa>
for businesses, losing labels/transactions may be far more harmful than losing some money
< wumpus>
the hd seed is in dumpwallet, for (2) a backupwallet makes sense
< wumpus>
if you just want to backup all data why not use the database format itself
< Provoostenator>
Yes, and businesses need a paper trail for audits, ideally one that doesn't contain a private key.
< sipa>
so perhaps there should be a way to separate the two
< Provoostenator>
Because a database format is too specific.
< jonasschnelli>
sipa: hardware wallets?
< wumpus>
Provoostenator: too specific for what? the metadata format is also completely specific to this wallet
< Provoostenator>
There's no bookkeeper / accountant in the world that can handle a .dat file, but they all know CSV or some other more text-like standard.
< jonasschnelli>
I think long term we should not expect that private keys are on the same machine then bitcoin core runs (at least not with the current one process design)
< wumpus>
Provoostenator: the GUI can do a csv export of transactions
< wumpus>
Provoostenator: if that's what you want
< wumpus>
also you can trivially implement that with a listtransactions then convert the JSON to CSV
< wumpus>
no need for that to be the client's storage format
< wumpus>
too many people are trying to solve problems by changing the program's internal storage format to be an external interface
< wumpus>
that's IMO wrong
< Provoostenator>
Ah I didn't know that feature. That's a good step and probably the most important metadata.
< wumpus>
if you want to export something, export it somehow, export exactly the data you need for some reason
< wumpus>
doesn't need to map to any internal data storage separation
< Provoostenator>
There's also the issue of long term storage.
< wumpus>
do you care how mysql stores its files? (besides it being efficient)
< Provoostenator>
In 50 years a txt dump will be readable, I doubt anyone can still parse the DB format.
< wumpus>
Provoostenator: that's where the dump format is for!
< Provoostenator>
@wumpus true
< wumpus>
it contains the private keys and the HD seed
< meshcollider>
And if you go and review #11667 then the dump format can include the scripts too ;)
< wumpus>
we have a surprising lot already covered with the current functionality
< sipa>
i wish we didn't have to continue the "bag-of-keys-and-script" approach in dumps, but i don't think there is a way around it now
< wumpus>
how do you mean? how would a dump work if it doesn't contain keys and scripts?
< sipa>
wumpus: read my writeup
< Provoostenator>
@sipa apart from your (useful) writeup, is there any other good documentation on how the wallet database and in memory storage currently works?
< wumpus>
at least for compatiblity with other software it's probably useful if it contains all that data
< bitcoin-git>
[bitcoin] ldm5180 opened pull request #11760: [crypto] Refactor HMAC_SHA to eliminate code duplication (master...generic-hmac_sha) https://github.com/bitcoin/bitcoin/pull/11760
< * jonasschnelli>
not sure if new contributors should fiddle with the SHA256 code
< Varunram>
Hey, I'm new here but thanks for the attention to new devs, it'll help a lot!
< Varunram>
I'd like to pitch in regarding GSoC, applications open january 4, if core is interested. You guys will be required to propose a project (or at least a list of possible projects) and applicants will have to choose from them. First time orgs get only 1-2 slots though
< Varunram>
Doesn't matter for a big project like Core, but still, my 2 sats :)
< wumpus>
Varunram: thanks for the info - but yes we'll have to think a bit about possible projects then,maybe a topic for next meeting
< Provoostenator>
wumpus: so IIUC: SPV uses more bandwidth than the just-send-pubkeys approach, but doesn't require running an electrum server?
< Provoostenator>
jonasschnelli: and IIUC your goal with #10794 is to pave the way to run a Core wallet in SPV mode?
< jonasschnelli>
Provoostenator: SPV (especially full block without client side filtering) takes much more bandwith...
< Provoostenator>
I should have more specific: to use bloom filters?
< jonasschnelli>
Electrum does not preserve your privacy
< jonasschnelli>
Bloom filter also not
< jonasschnelli>
*filters
< Provoostenator>
Well, it does if it's my own machine :-)
< jonasschnelli>
Yes.. and if the channel is encrypted and authenticated.
< jonasschnelli>
BF would be okay if BIP150/151
< Provoostenator>
I'm trying to think about the use case where I have my own full node in some place, but I want to make transactions on my computer / phone / wherever.
< Provoostenator>
And understanding the various trade-offs.
< wumpus>
electrum uses TLS by default, FWIW
< jonasschnelli>
Provoostenator: I see two solutions for that...
< Provoostenator>
(assuming BIP150/151 for the sake of argument)
< jonasschnelli>
a) you use BIP150 (or other enc/auth) via p2p and use SPV BF on your phone/remove client
< wumpus>
as long as you use it with your own server it's ok, and it already exists
< jonasschnelli>
b) you add a script to your bitcoin core machine that would server over TLS (an RPC proxy)
< jonasschnelli>
b2) while your remote phone is just an "extended" RPC client
< jonasschnelli>
(which would also have the private keys)
< Provoostenator>
An additional contraint is that I would trust the node for giving me correct data, but not for holding private keys. I'm not sure if that's a reasonable contraint.
< jonasschnelli>
Yes. The probably simplest approach would be SPV BF over auth/enc
< jonasschnelli>
10794 follows also another goal..
< jonasschnelli>
What if you don't have a remote node?
< jonasschnelli>
10794 (and future work) does allow you to use the wallet while your node is still bootstraping
< jonasschnelli>
My primary goal is to work against the current wallet trend... which is..
< Provoostenator>
@jonasschnelli b2 might be acceptable with an ecrypted wallet, but a seems better
< jonasschnelli>
centralized validation, and even remote key holding
< jonasschnelli>
The current bitcoin wallets do loose one of the primary elements Bitcoin can defeat "Trusted third parties are security holes".
< Provoostenator>
I'm thinking e.g. a full node at home, where if someone physcailly breaks in I'd know about that and just not use it.
< jonasschnelli>
I'd like to see more users using Bitcoin Core as a wallet
< Provoostenator>
@jonasschnelli me too, but I think a more realistic scenario is more people running Bitcoin Core nodes and connecting their favorite wallet to it.
< jonasschnelli>
But right now,... the burden is just to hight
< jonasschnelli>
Provoostenator: both is possible....
< Provoostenator>
Although with things like #11720 it might be possible, certainly with bloom filters.
< jonasschnelli>
BIP150/151 would work towards trustworthy direct connections
< jonasschnelli>
Provoostenator: Sure!
< Provoostenator>
Right, these are all useful ingredients. I'm mostly trying to wrap my head around how they all fit together.
< jonasschnelli>
With the current RPC calls it would also be possible to write a (iOS) client that would speak over RPC (via a proxy/apache script via TLS)... the client would hold the keys
< jonasschnelli>
and use fundrawtransaction and watch-onlies
< Provoostenator>
Something tells me more people will use it if it "just works" and everything happens on the phone.
< Provoostenator>
Another benefit of using the Core code base is that you don't have to re-invent the wheel for things like coin selection (especially if it gets dramatic improvements in the future).
< jonasschnelli>
The "just works" approach is very important and a reason why I try to kick BIP150/151 forward even with the fact that it's already sort of possible with stunnel, VPN, TOR
< wumpus>
TIL there's a program called "tig" that is a ncurses (terminal) git UI, I really like it
< * jonasschnelli>
executing "brew install tig"
< jonasschnelli>
looks nice
< wumpus>
yes I'm surprised I hadn't heard about it before
< Provoostenator>
Speaking of tools: any favorite IRC clients for OSX? And a good way to setup email notifications if someone mentions you when you're offline? I'm trying to set that up through the Slack bridge now.
< sipa>
Provoostenator: i use ssh + screen + irssi
< wumpus>
dunno about mail notifications, but I kind of like quassel, it has a separate frontend and backend, so from whatever device you log in your backlog is there, including highlights if someone mentioned you
< jonasschnelli>
Provoostenator: I use Textual 7 (macOS) with a ZNC bouncer
< jonasschnelli>
Provoostenator: I use a ZNC mod that sends me a Telegram on PM
< wumpus>
there's also a pyquassel to connect programmatically, it's possible to watch for messages and set up things like mail notification or other scripting
< jonasschnelli>
(the mod has various push channels)
< jonasschnelli>
Provoostenator: ZNC is you IRC swiss army knife... also can log for you, etc.
< wumpus>
but yeah you can do the same with irssi - there's even an irssi based frontend for quassel if you want to combine them :-)
< Provoostenator>
Thanks, I'll take a look at both approaches.
< wumpus>
github's commit notifier is broken again
< jonasschnelli>
wumpus: the twitter bridge worked... only IRC?
< wumpus>
jonasschnelli: seems so!
< phantomcircuit>
wumpus, i like znc more than quassel
< wumpus>
ok, I prefer quassel
< sipa>
real programmers use telnet, and speak RFC2812 natively
< wumpus>
hah
< phantomcircuit>
sipa, gl replying to pings in time from freenode
< telnetuser>
@phantomcircuit no problem, you'll see!
< wumpus>
phantomcircuit: no more idlers
< phantomcircuit>
lol you are using telnet
< sipa>
damn, how do i type a CTCP version reply? :(
< phantomcircuit>
sipa, gotcha
< phantomcircuit>
you have to be able to send 0x01