< sipa>
i may not be around for the meeting tomorrow, but BlueMatt suggested earlier that maybe instead of using /16 groups for the IP range "groups" (which affect outgoing connections, and addrman bucketing), we could use actual AS numbers (the identifiers of the autonomous networks which can assign IP addresses)
< sipa>
so i spent some time coming up with a compact encoding for a table with all AS numbers, and it seems a binary trie that's stored as a concatenation of variable-length integers with byte offset jumps can encode the whole thing (IPv4 and IPv6) in about 2 MB
< sipa>
does that seem like a reasonable thing to build into bitcoind, or ship as a file along with it?
< sipa>
if too large, we could of course approximate things by merging adjacent tiny IP ranges
< phantomcircuit>
sipa, where would you get the as information?
< sipa>
phantomcircuit: BlueMatt
< phantomcircuit>
BlueMatt, ^
< midnightmagic>
just a snapshot of the current bgp routing table aint it
< bitcoin-git>
[bitcoin] shannon1916 opened pull request #16199: rpc: fix showing wrong amount when not all inputs are from me in gettransaction (master...master) https://github.com/bitcoin/bitcoin/pull/16199
< sipa>
midnightmagic: yeah
< midnightmagic>
a feed of bgp routing changes might be helpful on its own for checking connection flapping -- and especially bgp hijack detection
< midnightmagic>
there's a twitter account that publishes such things too, it's pretty eye-popping the AS who end up stealing other peoples' AS.
< midnightmagic>
Its AS database isn't updated though so sometimes it looks like nobody hijacked anything but of course, it ends up being like telstra hijacking like .. Comcast or something
< bitcoin-git>
[bitcoin] laanwj opened pull request #16201: devtools: Always use unabbreviated commit IDs in github-merge.py (master...2019_06_ghmerge_unabbrev) https://github.com/bitcoin/bitcoin/pull/16201
< wumpus>
promag: it is, though the initial build uses the cache for the branch (at least I hope so)
< wumpus>
it *used to*
< wumpus>
but I'm starting to think something changed in regard to caching
< wumpus>
fanquake: ahh so it's another case of "Initial build successful, but not enough time remains ..." I wish that was displayed at the top instead
< bitcoin-git>
bitcoin/0.17 05fb9f7 MarcoFalke: build with -fstack-reuse=none
< bitcoin-git>
bitcoin/0.17 b5a4abe Pieter Wuille: Add test for GCC bug 90348
< bitcoin-git>
bitcoin/0.17 c165df1 Wladimir J. van der Laan: Merge #16163: 0.17.2 backport: build with -fstack-reuse=none and test cas...
< bitcoin-git>
[bitcoin] laanwj merged pull request #16163: 0.17.2 backport: build with -fstack-reuse=none and test case (0.17...backport_stack_reuse_none_017) https://github.com/bitcoin/bitcoin/pull/16163
< promag>
MarcoFalke: thanks for the backport
< jonasschnelli>
#proposedmeetingtopic travis stability and usefulness in current state
< MarcoFalke>
So why are we using tinyformat for printf, but not for fprintf et al?
< MarcoFalke>
Also, it seems like we might not be using tinyformat at all, since it depends on include order what the symbol refers to, no?
< MarcoFalke>
re travis: I have deleted all caches on travis
< MarcoFalke>
It can't get any worse than now
< jonasschnelli>
I think we just use it for log printing,.. right?
< MarcoFalke>
Yeah, why. Why not for printing to a file handle
< MarcoFalke>
stdout or whatever
< jonasschnelli>
Maybe due to security/stability concerns? but probably not
< wumpus>
tinyformat should ideally be used for everything, but at the moment it's used for 1) formatting to strings (strprintf) 2) log formatting
< wumpus>
(LogPrint / LogPrintf)
< MarcoFalke>
it is also used for some printf, depending on include order
< wumpus>
I strongly disagree with "we might not be using tinyformat at all", in contrast to libc printf implementations, tinyformat is type safe
< MarcoFalke>
since the tinyformat header has a printf in it
< MarcoFalke>
I meant "we might not be using tinyformat *accidentally* at all"
< wumpus>
we had some really awful problems with platform libc implementations (esp windows), with regards to strings, locale, even formatting 64 bit integers, there's no way we're going back to that
< wumpus>
ooh
< MarcoFalke>
because it might depend on include order
< wumpus>
is bare 'printf' used anywhere?
< wumpus>
it shouldn't really be
< jonasschnelli>
I don't think we use bar printf anywhere
< jonasschnelli>
(expect ctaes bench)
< MarcoFalke>
oh, yeah
< MarcoFalke>
But we use fprintf
< jonasschnelli>
*except
< wumpus>
(it's kind of useless as stdout might be redirected to nowhere, ther's some fprintf(stderr, ) where we're sure it's before the daemonize I guess, would be OK with replacing those with something that uses tinyformat)
< wumpus>
but please don't redefine fprintf for that but define a new function
< MarcoFalke>
ok, makes sense
< wumpus>
it's kind of bizarre that tinyformat redefines printf
< wumpus>
I think you have agood point in that
< wumpus>
we definitely should avoid using that one
< jonasschnelli>
"Tinyformat: A minimal type safe printf replacement"
< MarcoFalke>
Yeah, I was really confused when reading through all this after I had issues with the %zu format specifier on windows
< MarcoFalke>
I dig all the way to #326 :(
< wumpus>
jonasschnelli: yes, that doesn't mean the function has to be called that :)
< gribble>
https://github.com/bitcoin/bitcoin/issues/326 | Remove the IRC triggered hard last-seen check from the peer selection. by gmaxwell · Pull Request #326 · bitcoin/bitcoin · GitHub
< bitcoin-git>
[bitcoin] emilengler opened pull request #16204: Change Content-Type to plain text for the webroot (master...patch-emilengler) https://github.com/bitcoin/bitcoin/pull/16204
< BlueMatt>
i dunno where they pull it, but I didnt try to correlate between v4 -> RIR objects -> asns
< BlueMatt>
which, I suppose, may also be a relevant approach
< BlueMatt>
phantomcircuit: the nice thing is what sipa wrote will eat bird "show route all" output and map it, so its rather easy to get others' dumps as well
< phantomcircuit>
BlueMatt, that is nice
< phantomcircuit>
it's definitely better to do asn grouping than /16
< nkohen>
Hey all, I've implemented golomb coded sets as mentioned in BIP 158 and I'm currently trying to implement serialization of blocks into block filters (from the same BIP) and I'm looking over how core does this and it appears to me that entire output scripts are being put in blocks and not just the data pushed on the stack as specified in BIP 158. Is this true?
< sipa>
> The scriptPubKey of each output, aside from all OP_RETURN output
< phantomcircuit>
nkohen, the same data is probably in the filter more than once in different forms
< sipa>
phantomcircuit: no
< sipa>
it's the scriptPubKeys of all outputs in the block, and the scriptPubKeys of all outputs spent by the block
< jonasschnelli>
nkohen: core has two filter types
< sipa>
jonasschnelli: wut?
< sipa>
no it doesn't
< jonasschnelli>
no...
< jonasschnelli>
:(
< jonasschnelli>
only basic is imlemented
< jonasschnelli>
I though since we had that option the extended was also there...
< sipa>
the extended one was dropped from the bip
< jonasschnelli>
nkohen: maybe lookup the function "BasicFilterElements" in blockfiler.cpp
< nkohen>
jonasschnelli: that is the function I've linked to, and I'm not super fluent in c++ might be the issue but it looks like we call `elements.emplace(script.begin(), script.end());` where script is a CScript, and unless that somehow skips the opcodes and such which aren't data it looks like they get added?
< sipa>
nkohen: no
< sipa>
the spec says the entire scriptPubKey is added to the filter and nothing else
< sipa>
not sure to what extent we decided that the high priority list should have an implicit "basic concept ack" because discussed in the meeting; but to the extent that it is, i think the list entries are great :)
< MarcoFalke>
jtimon: are you adding that as "Chasing Concept ACK"?
< MarcoFalke>
wumpus: Yeah, it has been open for months and hasn't received any attention
< fanquake>
dongcarl do you have any build related blockers? Guix or otherwise.
< sipa>
concept ack
< meshcollider>
concept ack for adding Marco's PR
< dongcarl>
fanquake: looking
< sipa>
agree on jtimon
< sipa>
agree on jtimon's pr as chasing concept ack
< jnewbery>
I'd like #16060 on there. I have a couple of review comments from last week to address. It's blocking some work that BlueMatt is doing on reducing cs_main usage
< meshcollider>
the EOL date for 0.16 seems fine though, just under 1 year was the rule I used last time I updated it
< cfields>
Again, though, can we somehow automatically keep this in sync with SECURITY.md? I'm afraid of them drifting apart.
< wumpus>
yes
< fanquake>
We could just make security.md the place where they live, and they get manually/automatically pulled out of their for website updates?
< meshcollider>
+1
< wumpus>
so the information in security.md is only for security fixes, not general maintenance
< wumpus>
it should list the versions that are not entirely EOL I guess
< cfields>
fanquake: +1
< meshcollider>
Kinda makes more sense to have it in the repo though than on a separate website?
< wumpus>
maybe ...
< wumpus>
I'm usually not in favor of importing everything into the repo
< achow101>
it should be possible to have SECURITY.md fetched and parsed during the build for the website. I know bitcoin.org does something similar to generate the contributors page
< nehan>
fwiw: the motivation for SECURITY.md was in part because people who wanted to report things did not know about bitcoincore.org
< wumpus>
like it's fiiine to maintain different things in different places, it doesn't all have to go through the same bottleneck you know
< wumpus>
nehan: yes, for reporting it makes perfect sense
< sipa>
i think the SECURITY.md living in the repo itself is weird, as it's not tied to the specific source tree it lives in (this is also true for release notes etc)
< MarcoFalke>
Add it to "release-process.md"? *ducks
< meshcollider>
Fair enough actually yeah
< MarcoFalke>
sipa: Release notes should be shipped, so at least during build you'd have to fetch them
< wumpus>
sipa: I think the reporting emails and gpg keys are
< cfields>
sipa: elaborate?
< wumpus>
sipa: as for the schedule, I guess I'd somewhat agree
< sipa>
cfields: which versions are still under maintenance, eg
< wumpus>
but that wasn't the main point of security.md, it was to let people know where to report critical security issues
< fanquake>
security.md is also being removed from any branch that isn’t master right.
< sipa>
if someone randomly clones the repo, leaves their copy untouched for years... someone stumbles upon it (like nkohen's with roasbeef's repo above ^), and sees the version they think they
< wumpus>
and I guess what releases to report them for ? like, if you find a problem with 0.8 now it's probably not useful to report it
< MarcoFalke>
So lets remove the eol dates from there again?
< sipa>
if someone randomly clones the repo, leaves their copy untouched for years... someone stumbles upon it (like nkohen's with roasbeef's repo above ^), and sees the version they think they're working with is maintained
< cfields>
wumpus: fair enough.
< sipa>
i don't know if this confusion doesn't weigh up to having the information in a well-visible location
< cfields>
MarcoFalke: I'd +1 that as well. I just don't like the idea of having them maintained separately in two places.
< wumpus>
sipa: yes
< wumpus>
ok, let's remove the dates from security.md
< MarcoFalke>
#action remove the dates from security.md
< sipa>
we could have a link in security.md to the authoritative place where that information is maintained
< instagibbs>
ack
< meshcollider>
+1
< fanquake>
If that’s done. Second proposed topic is where to have the next Core Dev meetup.
< cfields>
sipa: +1
< moneyball>
hi
< MarcoFalke>
ok,next topic
< wumpus>
#topic when/where should we have next CoreDev? (moneyball)
< moneyball>
Thanks for those that attended CoreDev. By all accounts I think it was a success!
< wumpus>
agree!
< MarcoFalke>
Once per release seems fine to me
< moneyball>
We had 18 responses to the feedback survey. The summary is "don't change anything." 72% want to keep reviewing/merging unstructured. 93% liked the scheduled talks. 83% think 3 days is the optimal duration. And majority of people like the # attendees and the mix.
< meshcollider>
Definitely, thank you \i/
< cfields>
moneyball: thanks for organizing!
< moneyball>
If anyone has other feedback please either fill out the form or let schmidty or me know.
< instagibbs>
I definitely think it was the best so far
< jonasschnelli>
Yeah. Thanks moneyball!
< sipa>
indeed!
< fanquake>
\o/
< sipa>
also congrats on getting Gary fired
< moneyball>
thx :)
< moneyball>
So I started thinking about the next CoreDev already! Aiming for 6-9 months from now which means December to March. Considerations include, most importantly, what % of active Core devs can attend, and after that, minimizing aggregate travel time/timezone shift/cost of travel, exploring new locations, winter weather, ease of organizing (eg piggyback off of conferences or have local Core devs to help)
< jonasschnelli>
indeed. Congratz moneyball
< MarcoFalke>
Who is gary?
< cfields>
moneyball: since there's typically a large turnout for Financial Crypto, maybe we could piggyback there?
< cfields>
It's usually within that window.
< wumpus>
not December please :)
< moneyball>
Initial candidates include SF, Austin, NYC, Uruguay mid-December (La Bitconf), and Australia (possible Feb 2020 Bitcoin conf). If anyone has other candidate dates/locations please let me know! I'll likely send a survey around (sorry for all the surveys but it is really helpful!) allowing Core devs to rank candidate locations.
< instagibbs>
internet kinda really sucks there cfields IIRC
< jonasschnelli>
We haven't had a CoreDev in Hawaii...
< nehan>
moneyball: also MIT Bitcoin Expo
< fanquake>
Marco A fake SquareCrypto intern.
< MarcoFalke>
Ah, thx. I uninstalled twitter
< moneyball>
nehan i will add that to the list!
< instagibbs>
Holiday season is a prob a no-go, as wumpus said.
< sipa>
cfields: i'd personally like a coredev colocated with FC, but i'm not sure it's that accessible for everyone (especially people who don't have company funding for travel)
< jnewbery>
financial crypto is 5 days. piggybacking would make a very long trip for people
< cfields>
sipa: good point.
< achow101>
moneyball: there's also advancing bitcoin in london in feb
< instagibbs>
co-located to another less-academic conference also has the benefits of people can submit talks/run workshops and piggyback that
< jonasschnelli>
coredev.tech does fund travel expenses of those not receiving support from employers
< sipa>
about Boston: not everyone can enter (or likes to enter) the US
< fanquake>
I’d probably almost strike Aus of the list for Dec - Feb. Too hot.
< sipa>
fanquake: i was very confused for a minute and thought you were talking about austin
< instagibbs>
dropbears don't like the heat though
< jnewbery>
fanquake: maybe in Binnu. There are places that aren't quite as hot
< fanquake>
Yes but neither do Core Devs I’d assume.
< fanquake>
jnewbery if your happy with 40C days, go ahead. We’ll have Core dev on the beach.
< meshcollider>
NZ is a bit more mild than aus, come over here instead ;)
< jnewbery>
I meant maybe Binnu was too hot and we could have it somewhere else
< wumpus>
hehe
< fanquake>
ah right
< * sipa>
would suggest Iceland (nicely in the middle between EU and US), but only if in summer :)
< moneyball>
regarding not wanting to enter the US, i would like to better understand how big of an issue this is. if those affected are comfortable, please reach out to me and let me know so it is taken into account
< wumpus>
Iceland :D
< jonasschnelli>
ack Iceland
< instagibbs>
there are >=2 people that came to last coredev that can't come generally speaking.
< moneyball>
yeah the next next CoreDev is already decided...summer 2020 iceland :)
< nehan>
i was also going to suggest iceland
< instagibbs>
It should be over the next halving tbh :)
< fanquake>
moneyball I think this could be left for suggestions/comments. Anything else you wanted to bring up? Otherwise final proposed topic is Travis instability.
< moneyball>
all good thanks for the input everyone
< meshcollider>
Is there a way to provide input that's easier for you? Like a survey of our preferred location and timing or something?
< cfields>
moneyball: fyi, the last thing done at FC is voting on the next location. Which is cool because the people there are obviously a great sample.
< cfields>
Just throwing that out there because I think it's neat :)
< sipa>
it's just an informative vote though; the actual decision is made much later
< cfields>
right, that.
< wumpus>
heh it doesn't include the people that weren't able to come
< meshcollider>
Confirmation bias lol
< sipa>
self-echochambering is best echochambering
< wumpus>
#topic Travis stability and usefulness in current state (jonasschnelli)
< wumpus>
travis is failing sooo much on PRs now
< moneyball>
i'll do a survey ranking locations
< jonasschnelli>
We probably discussed that already to a certain level. But somehow I think we should have a plan how to get travis back on track
< jonasschnelli>
Right now on an avg of 3-4 tries travis will succeed.
< wumpus>
it's absurd that even PRs that change like 3 comment lines fail
< cfields>
jonasschnelli: by travis do you mean Travis the service? Or our c-i checks in general?
< wumpus>
I know it's bad but I'm already starting to effectively ignore it
< jonasschnelli>
It can't be that we just need to re-trigger builds all over... this makes the value of CI partially disappearing
< fanquake>
cfields: it seems like lots of invalide caches, casting constant depends rebuilds? At least what I’ve seen.
< wumpus>
it used to be 'oh this PR fails, it needs more work by the author' but it's never the author's fault !
< fanquake>
*invalid, *causing
< wumpus>
so I run the tests locally, which is incomplete but at least reliable ...
< achow101>
can drahtbot be made to retrigger travis on certain kinds of errors?
< jonasschnelli>
I think we should have a strategy how to escape this mess...
< achow101>
otherwise maybe we should consider using a different CI
< jonasschnelli>
eventually... but that is a lot of work
< fanquake>
The depends builds then obviously take up all the build time, and the build dies.
< wumpus>
yes, we need something better
< cfields>
Maybe we should finally consider decoupling the depends builds?
< wumpus>
another CI would be ok with me
< jb55>
circle-ci ?
< sipa>
are there any self-hostable CI systems that integrate with github?
< cfields>
(afaik I've been the one arguing for keeping depends compiles included in the past, but things have changed...)
< wumpus>
I liked travis a lot when it worked but this is just not doable anymore
< jonasschnelli>
I think it would be worth if willing people can propose a researched alternative on next meeting
< wumpus>
cfields: you mean, using distro deps instead of depends?
< fanquake>
I think those would be interesting to review.
< jonasschnelli>
if we have volunteers...
< wumpus>
cfields: too bad travis has such old versions
< meshcollider>
sipa: Jenkins is self hostable I think
< wumpus>
it's still trusty isn't it?
< cfields>
wumpus: I mean automating depends builds separately, and having bitcoind builds fetch pre-built depends binaries.
< wumpus>
cfields: ohh makes sense
< wumpus>
cfields: then only do depends rebuilds, if depends change
< fanquake>
FYI if anyone is unaware, we already have a circle CI confit in the repo, and Marco runs its against his own repo I’m pretty sure. Has a PR open for confit updates atm.
< bitcoin-git>
[bitcoin] instagibbs closed pull request #15796: CReserveKey should not allow passive key re-use, debug assert in destructor (master...burn_reserve) https://github.com/bitcoin/bitcoin/pull/15796
< MarcoFalke>
Yeah sorry for missing the last topic
< MarcoFalke>
But it seems travis caches are broken in some way
< MarcoFalke>
Still investigating ...
< fanquake>
#proposedmeetingtopic Requiring 2FA for bitcoin orgs on GH. This was discussed a little at Core dev, and we got 4-5 members to turn 2FA on.
< jamesob>
fanquake: sorry, was afk. will tap matt next I see him
< dongcarl>
fanquake: Probably 2FA but no SMS recovery?
< hebasto>
SMS recovery seems to be a great security issue
< fanquake>
dongcarl hebasto correct. kanzure might have some thoughts to share on that.
< phantomcircuit>
dongcarl, there's no way to configure github organizations to require 2fa but not sms
< dongcarl>
phantomcircuit: :-/
< phantomcircuit>
dongcarl, and yes sms is basically useless
< wumpus>
huh I definitely set up 2FA but not SMS recovery on my github
< wumpus>
that was a long time ago though, maybe it changed
< kanzure>
oh, yeah, we should enforce that users can't turn SMS recovery back on
< meshcollider>
It's possible to have 2FA without SMS but he means there's no way to enforce it, from an organization setting
< meshcollider>
GitHub thinks 2FA is 2FA
< meshcollider>
We could also discuss removing some inactive members from the org which was briefly brought up in discussion too
< phantomcircuit>
achow101, can we not use acronyms for class names
< phantomcircuit>
it makes it much more confusing for people who aren't looking at the code everyday
< phantomcircuit>
wumpus, yes you can set that up on your personal account, but an organization cannot enforce it
< achow101>
phantomcircuit: where?
< phantomcircuit>
achow101, the wallet class stuff
< phantomcircuit>
SPKManager
< phantomcircuit>
I know it's longer to type and that's kind of annoying without autocomplete, but it's definitely more confusing
< phantomcircuit>
maybe people disagree with me, but that's what i find looking at stuff i haven't looked at in a while