< wumpus>
dongcarl: no, the name proxy is not for onion addresses, it is for DNS names where the network is not known, I think it ever only had one use (but I can be wrong) connecting to the DNS seeds for one-shot. It's kind of a hack and would be happy if it's no longer used and can go...
< wumpus>
gmaxwell: thanks!
< wumpus>
dongcarl: but last time I tried to I couldn't remove it because in the "fully behind behind tor proxy" case the one-shot connections were the only way to initially seed peers due to lack of DNS resolve support, I remember cfields_ was working on changing this flow at some point but not the details at the moment
< wumpus>
(I think in practice it might be valid to replace use of the name proxy by use of the IPv4 one, as they're always the same)
< dongcarl>
Oh I'm guessing the last time you tried to remove this, bitcoin only had support for SOCK4 and couldn't resolve thru the normal proxy?
< gmaxwell>
my vague recollection is that we can never _resolve_ via a proxy, we can _connect to a name_ but conecting to a name isn't sufficient to get us the results of a dnsseed query...
< wumpus>
no, I don't think so--it has always only supported SOCKS4A and SOCKS5 which both support name addresses
< wumpus>
exactly ^^
< wumpus>
that's the problem, you can connect to a name but you never learn the underlying address
< gmaxwell>
so what one shot does is just connects to a name, gets addresses from it, and disconnects. (we don't want to throughly bias our connectivity to whatever unlucky hosts are in the dns seed output, as it risks overloading them, esp since some resolvers don't round robbin well)
< dongcarl>
Ah I see, so the proxy does the lookup and you don't get any transparency in that
< wumpus>
yes
< dongcarl>
gmaxwell: oh like connects to a bitcoin node and briefly get some other peers we can connect to from it then say bye?
< wumpus>
right
< wumpus>
gmaxwell: for me it would be convenient if banlist.cli.txt had ${BITCOIN_CLI} or something on every line instead of ./bitcoin-cli, and allowed it to be overridden, for example to point to a custom bitcoin-cli path or providing a -datadir/-conf argument
< wumpus>
(it could still default to ./bitcoin-cli ofc)
< * luke-jr>
wonders how many people just curl and run the script blindly
< wumpus>
I definitely don't run it blindly
< gmaxwell>
luke-jr: certantly wasn't my intent, at least the way I use it is copy and paste, so not blindly.
< luke-jr>
I didn't mean to imply anyone *here* would run it blindly :x
< gmaxwell>
luke-jr: perhaps I should make the first line of the file a sleep 10000000
< luke-jr>
heh
< luke-jr>
do modern x86 systems still have PC speakers? maybe make it play a rickroll before that ;)
< echeveria>
gmaxwell: if you want to be google. while true; do sleep 1000000; done. that was the standard format for bypassing vulnerabilities in Internet Explorer.
< TD-Linux>
luke-jr, they do, but it may be emulated in various ways (maybe even with smm)
< sipa>
/me has fond memories of the PLAY command in GW-BASIC
< * luke-jr>
has Linux software to emulate the BASIC PLAY command <.<
< luke-jr>
turns out there's actually a spec for it called MML
< emilr>
speaking of seeds/peers I have bitcoin running in an isolated environment: it can't connect to anything except .onions and it has some peers added with addnode, the behaviour that I'm seeing is that bitcoind does not learn about other peers from its existing ones, possibly because dns seeds fail, the expected behaviour is that bitcoin learns about other peers from it's existing ones (the ones
< emilr>
added with addnode, connect has the same behaviour)
< emilr>
wumpus: sed -e 's|./bitcoin-cli|ANYTHING -you -want|g' banlist.cli.txt
< gmaxwell>
emilr: why do you say that it doesn't learn
< emilr>
peers.dat is 97K so basically empty and it doesn't connect to any other peer except the ones added with addnode
< gmaxwell>
97k is far from empty there aren't that many onion peers.
< gmaxwell>
so that part isn't interesting.
< gmaxwell>
any chance you have automatic connections disabled? how is "it can't connect to anything except .onions" accomplished?
< emilr>
onlynet=onion and bitcoin user is blocked from making connections to the clearnet
< emilr>
also tor is configured with SocksPort 9050 OnionTrafficOnly
< gmaxwell>
emilr: is it logging attempts to connect to things?
< emilr>
gmaxwell: I was under the impression it would fetch all peers regardless of net but it did indeed discover some peers, I see 190 different .onions in two weeks, I think what happened is that bitcoin gave up on a lot of those peers after few failures but failure to connect is a normality in the tor land
< emilr>
I'll increase -timeout and wipe peers.dat to see if it makes much of a difference, I guess bitcoin needs to be more agressive when it has a limited number of peers
< gmaxwell>
emilr: if it's onlynet=onion it'll only store onion peers
< wumpus>
right, it discovers only .onion if that't the only enabled network
< emilr>
makes sense, although other onlynet=tor nodes I ran had clearnet peers as well but those nodes weren't so isolated from clearnet
< DeanGuss>
onlynet=onion is the only thing that works, not onlynet=tor -- is that why maybe it is ignoring your onlynet=tor ?
< wumpus>
if onlynet=tor have clearnet peers (even tries to connect to anything non-clearnet) that's would be a bug, it definitely isn't my experience on the ones I run; though if the listening port is open to the internet, even without it advertizing your address, there's a chance peers might find you by random
< wumpus>
onlynet=tor works too, it's equivalent
< DeanGuss>
huh. ok, I thought it didn't but I only tried a few releases ago
< wumpus>
it's deprecated and will give a "Warning: net name 'tor' is deprecated and will be removed in the future. You should use 'onion' instead." error but it's definitely not going to silently ignore it
< wumpus>
(onlynet isn't mentioned in doc/tor.md btw; would make sense to do so)
< kallewoof>
Does the encryptwallet feature in bitcoin core relate to bip 38 in any way, or do they just look similar? It looks like it is basically doing the same thing but with scrypt replaced.
< wumpus>
it does not relate to bip38 in any way; if anything, most people involved with bitcoin dev really dislike bip38 and it's per-key encryption
< jonasschnelli>
what could be the reason for a null leveldb obfuscation key?
< jonasschnelli>
Using obfuscation key for /btc/data/bitcoin/blocks/index: 0000000000000000
< wumpus>
jonasschnelli: the block index isn't obfuscated is it? just the utxo set
< jonasschnelli>
wumpus: Oh. NM. I missread it (was confused by an empty key)
< kallewoof>
wumpus: Oh, okay. Thanks
< wumpus>
kallewoof: I think the only commonality is that it both involves some kind of loop for key-stretching
< kallewoof>
wumpus: i finally spotted the "unanimously discouraged" part in comments and read thru the comments page on the bip. I kind of just want to combine a key and a user password into a password-protected-key. Seems the encryptwallet feature does basically that, though, unless I'm confused.
< wumpus>
it uses a standard mechanism at least
< wumpus>
BIP38 has some really goofy things, it was never really peer-reviewed just dropped out there,but I don't remember what anymore-please don't use it, though
< kallewoof>
I won't! Thanks
< emilr>
the good news is that you can bootstrap bitcoind in a tor only environment from the hardcoded onion seeds, the bad news is that you can't using a rpi :) unless you use connect, it appears that the latency of the usb disk adds to negociation time thus making peer discovery unworkable, a sad day for the poor little guy
< wumpus>
huh, I believe we didn't update the hardcoded seeds for 0.18
< wumpus>
emilr: it shouldn't hold up network negotiation on disk access, does it time out on something?
< emilr>
most connections timeout at version handshake, takes about 10s for verack plus network latency, if bitcoin tor nodes run with the default timeout then you're out of luck
< wumpus>
I've never noticed that myself, just that block validation is slow, the network stuff seems to run at normal speed on my ARM boards
< wumpus>
(well, maybe slower, but not thus slow it's self-sabotaging)
< emilr>
3B+, I'm guessing it's not an issue if you don't sync over tor
< wumpus>
same with tor—though the difference might be that I always bootstrap from the DNS seeds, I haven't tried the built-in seeds for a long time
< wumpus>
I do
< wumpus>
but it's been a while, could try again
< wumpus>
first, seeds update for 0.18..
< bitcoin-git>
[bitcoin] laanwj opened pull request #15791: [do not merge] net: Hardcoded seeds update for 0.18 (master...2019_04_hardcoded_seeds_update) https://github.com/bitcoin/bitcoin/pull/15791
< wumpus>
ok something strange is happening here
< wumpus>
it looks like sipa's seed does return onion peers but somehow, makeseeds.py throws all of them out
< wumpus>
so I *guess* the problem is that onion peers, due to relative unreliability of the medium, get sorted lower and dropped out of the uptime scores
< wumpus>
but why this is suddenly an issue I don't know, maybe tor was down for a while on the crawler?
< emilr>
pubmsg: or due to the fact that the number of peers increased pushing onions even further down the list, I'll check
< wumpus>
maybe change the uptime requirement for onions?
< wumpus>
bah, even changing it to 25% with no other changes only gives 10 onions
< wumpus>
it's better than nothing, I guess
< emilr>
pubmsg: I think the whole process needs a bit of rework, I wouldn't trust whatever list the script generates
< emilr>
pubmsg: version 17.0 has 120ish onion peers, this one generates just 2, theres no other checks for onion and ipv6, the input list is ripe with light clients and scanners, the uptime goes back only 30days which makes it easy to get a majority of nodes into the seed list, I will dedicate some time to it next week
< bitcoin-git>
[bitcoin] jonatack opened pull request #15792: doc: describe onlynet option in doc/tor.md (master...add-onlynet-option-to-tor-docs) https://github.com/bitcoin/bitcoin/pull/15792
< wumpus>
emilr: I don't think that's very useful commentary, we need to know what step is causing problems, if the problem is with the script at all: if it's the DNS seed significantly under-reporting uptime for onion peers then it's not the script's fault at all
< wumpus>
in itself, 50% uptime is a reasonable requirement
< wumpus>
even using grep directly on the downloaded input I cannot corabborate your claim that "version 17.0 has 120ish onion peers"
< wumpus>
grep \.onion seeds_main.txt|grep "/Satoshi:0.17" -> one result
< wumpus>
unless you have a different source? if so, please let me know
< emilr>
sorry, what I meant is that bitcoin core 0.17 has 120 hardcoded onion peers
< wumpus>
oh!
< wumpus>
yes, I misunderstood your sentence then
< wumpus>
would make sense to do a scan on them and see how many are still functional, and if so, why the crawler is leaving them out
< emilr>
doing that now in order to see how many of the peers from 0.17 are still up, so that if you're pressured to release you can use the old list
< emilr>
wumpus out of 109 hardcoded onion seeds, 72 responded at first check, and none of them are in sipa's seeds, in fact only 3 out of the 109 hardcoded seeds are in seeds_main.txt which has 712 onions
< wumpus>
emilr: interesting
< emilr>
something is missing from the picture, sipa's seeds has all nodes since 2013, where are the hardcoded onions from if they're not in seeds_main.txt ?
< emzy>
I have no .onion
< wumpus>
heh
< emzy>
no idea what's the problem
< wumpus>
emilr: i don't think "has all nodes since 2013" is true
< wumpus>
for example the list doesn't include anything with zero uptime in last 30 days
< wumpus>
and that would be *many* if so
< emilr>
looks like it, once a node because unreachable it is not removed from the list, in fact most of the top 6500 servers by uptime were last updated in 12 Feb
< emilr>
wumpus: thats because once a server goes stale, it keeps it's uptime history
< wumpus>
in any case, for better or worse, everyting in the hardcoded seeds, ever, has come from sipa's crawler up to now
< wumpus>
I think that's acceptable because it's only a fallback
< emilr>
just check block height 2624 top nodes at 562647, and 4281 at 562646, that's the last time the list was updated
< wumpus>
emilr: did you configure the crawler with tor? (I haven't ever run a DNS seed myself so I don't know the details, unfortunately)
< stevenroose>
So is that a generic thing? Like all calls can do that?
< sipa>
i doubt that
< stevenroose>
Like gettxout, getrawtransaction, getblockheader?
< stevenroose>
Seems to come from CMerkleTx::GetDepthInMainChain
< sipa>
are you sure that pindex is the block the tx is included in here?
< sipa>
i think it's the block the _conflict_ is included in
< sipa>
just an orphaned transaction should have 0 confirmations
< sipa>
it's only when a doublespend of it gets confirmed that you get negative numbers
< sipa>
gettxout only applies to actually existing UTXOs, so negative confirmations don't make sense there
< sipa>
same for getrawtransaction
< sipa>
and it doesn't apply at all to blocks
< wumpus>
CMerkleTx is only used from the wallet; it'll be wallet calls at most that use that convention
< stevenroose>
sipa: srry didn't see your response. thanks. will report to Alekos, who bumped into this. I think he indeed double-spent his own tx after forking
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #15795: scripted-diff: Avoid name collisions in CChainState (master...1904-m_chain) https://github.com/bitcoin/bitcoin/pull/15795
< bitcoin-git>
[bitcoin] instagibbs opened pull request #15796: CReserveKey should not allow passive key re-use, KeepKey in destructor (master...burn_reserve) https://github.com/bitcoin/bitcoin/pull/15796
< gmaxwell>
sipa: did your crawler lose onion access? (see earlier discussion about nodes that are currently in the seeds list being no longer found by your crawler.
< luke-jr>
MarcoFalke: seemed like everyone agreed on the interpretation except sipa who took it literally at first, and didn't really follow up with an objection later *shrug* can ask sipa to be explicit I guess
< sipa>
luke-jr: i'm just unclear on the goal?
< sipa>
is the goal disabling because (a) they're experimental and someone doesn't trust the optimizations (b) because no assembler is available and the build system incorrectly detects that or (c) the compiler doesn't support the intrinsics
< gmaxwell>
disable machine specific crap that breaks weirdo compilers
< wumpus>
working around build problems, and analysis tooling, basically
< sipa>
the option was originally introduced because of (a), but i think that goal is gone
< gmaxwell>
b/c.
< gmaxwell>
yea, not a
< sipa>
if it's for b/c both, it shouldn't be called asm
< gmaxwell>
speaking of a, we don't enable asm for libsecp256k1 in the bitcoin core build, do we?
< wumpus>
in any case, if the configure checks worked 100% (so, it would disable those things automatically if the compiler didn't understand them) it wouldn't be needed I think
< wumpus>
gmaxwell: it does afaik
< sipa>
gmaxwell: pretty sure we do, but not the arm asm
< wumpus>
of course not the ARM asm :)
< wumpus>
#startmeeting
< lightningbot>
Meeting started Thu Apr 11 19:00:25 2019 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< jnewbery>
I'd request that #15750 gets considered for inclusion in the next rc. It completes a fix that is already in 0.18
< gribble>
https://github.com/bitcoin/bitcoin/issues/15750 | [rpc] Remove the addresses field from the getaddressinfo return object by jnewbery · Pull Request #15750 · bitcoin/bitcoin · GitHub
< jeremyrubin>
Hi! Also present
< instagibbs>
hi
< luke-jr>
jnewbery: that looks like removal of an API, not a fix; I don't really care though
< luke-jr>
better to remove in 0.18.0 than 0.18.1
< cfields_>
ah, sorry, irc went weird for me.
< cfields_>
See #15798, lots of useful info there. tl;dr: This is a cool project from Jeremy Rubin that allows us to use rust code from inside of Bitcoin Core. No plan yet, I mostly just wanted to spread the word that people should try it out and report back. It pretty much just works. It is surprisingly complete, but has only been tested in a few environments so far.
< wumpus>
jnewbery: looks like it's somewhat controversial, any reason to not do this for 0.19 instead?
< luke-jr>
anyway, CRust.. I don't think including Rust inside Bitcoin Core is a good idea, even optionally, so long as Rust requires trusting third party binaries
< MarcoFalke>
jnewbery: There shouldn't be any cost to only remove it in 0.19.0
< jonasschnelli>
Nice work jnewbery and cfields_!
< jonasschnelli>
luke-jr: it's an experiment AFAIK
< kanzure>
which third party library is that
< wumpus>
I really like the rust work
< jnewbery>
Because the deprecation warning in the RPC help text is removed in 0.18
< luke-jr>
jonasschnelli: experiments can be done outside Core
< cfields_>
luke-jr: there are people working on that. This is experimental. This discussion is not helpful at this time.
< kanzure>
you mean the rust binary?
< wumpus>
jnewbery: wha-why ?
< luke-jr>
kanzure: yes, rustc
< jonasschnelli>
luke-jr: read the PR desc
< MarcoFalke>
luke-jr: Bitcoin is experimental
< cfields_>
luke-jr: it says right there. NOT FOR MERGE.
< cfields_>
caps and everything :)
< jeremyrubin>
luke-jr: did you see Marco's bootstrapping link as well?
< wumpus>
it's an experiment, but it's good to have people aware of it
< cfields_>
anyway, I'd like to get some feedback from people who have actually used rust.
< instagibbs>
you'd want to ping #rust-bitcoin folks
< dongcarl>
Hi
< luke-jr>
jeremyrubin: as I understand it, mrustc only works on x86
< wumpus>
dongcarl: ^^
< jamesob>
rust is a pleasure to write - beyond that I don't have much input
< jonasschnelli>
I don't disagree with luke-jr concerns,.. but an experiments needs to start somewhere and Rust will evolve over time
< kanzure>
don't we also use other compilers in deterministic builds..?
< jeremyrubin>
luke-jr: can you not cross compile after bootstrap? Do you want to bootstrap on every platform?
< MarcoFalke>
Yeah, it is good to see that the build system changes are smaller than anyone expected
< jeremyrubin>
Anyways, as noted, this is just for enabling more experimentation
< cfields_>
luke-jr's complaint has been heard. Let's move on please.
< gmaxwell>
yeah, its not a for merge thing. Useful announcement.
< luke-jr>
kanzure: hopefully we are moving away from that
< dongcarl>
perhaps a good question is what parts of the code is memory safety the most critical
< wumpus>
anything that interfaces to the hostile outside world (e.g. P2P code)
< gmaxwell>
It would only be interesting to merge if there were some functionality written in rust that we wanted to include. It was surprising to me that it took as much as CRust in order to do that. (it isn't like we need anything special in the build system to link code written in C, for example)
< MarcoFalke>
The parts that are not going to be rewritten in rust any time soon ;)
< cfields_>
dongcarl: yes. also, what things are in-spec in rust that can only be undefined in c/c++.
< MarcoFalke>
dongcarl: ^
< instagibbs>
MarcoFalke, hah!
< jamesob>
a while back BlueMatt had talked about doing a secondary fallback network stack in rust which would activate if the existing p2p code started acting weird
< jamesob>
but that's obviously pretty speculative...
< wumpus>
that's a pretty neat idea
< sipa>
i don't understand the appeal of that idea
< sipa>
the complexity is all in the interaction between network code and the rest
< meshcollider>
I remember him bringing that up in tokyo
< jamesob>
two is one and one is none, I guess
< luke-jr>
gmaxwell: well, part of the concern is that people are more likely to write stuff others want in Rust than C++, with this; but maybe not a big deal if Rust is making good progress
< sipa>
not in the network code itself
< luke-jr>
jamesob: sounds more complex to detect "weird" than to just replace it
< jamesob>
luke-jr: could be
< cfields_>
sipa: I would think something like a rust-libevent would be pretty appealing. But yeah, that's a layer below us.
< wumpus>
it's not so much about the complexity I think, but potential safety issues, rust would be safer for the network code
< luke-jr>
if someone is compromising the original network stack, they can probably corrupt the alternate one too
< sipa>
wumpus: in a way that we've actually ever had problems?
< cfields_>
It's also worth considering that new tools could be written in rust. Doesn't have to be direct bitcoind integration.
< luke-jr>
rewrite the node around libconsensus? ;)
< gmaxwell>
IIRC. We have never had a bug in the network code that use of rust would have structurally prevented, we have however had many bugs of the same kind that occure in rust. see e.g. the somewhat recent parity (ethereum node software) network wide crasher vuln from memory exhaustion (I think ultimately stemming from livelocks)
< wumpus>
sipa: that we haven't noticed any problems doesn't mean that they don't exist, but sure...
< sipa>
iirc all scares we've had in the past were about unbounded data structures, processing interactions, buggy logic, ... not out of bounds issues or the sort
< wumpus>
a lot of this is to rule out entire bug classes, not just to fix bugs that we've found
< sipa>
wumpus: while introducing a complexity in making the two layers communicate
< gmaxwell>
at a cost of radically reducing the set of people who can review things, and more complexity from interfaces looking different on each side, however.
< wumpus>
rust fixes undefined behavior at least,we've had plenty of that reported
< jeremyrubin>
I think fixing some of the internal concurrent workers would be a good task
< jeremyrubin>
E.g., the CheckQueue or Task QUeue
< sipa>
my view is that if and when there is an actual project with useful code we'd like to introduce as a dependency, that happens to be written in rust, of course
< cfields_>
gmaxwell: for now.
< jeremyrubin>
As that can just be treated as a scheduler
< gmaxwell>
Put it this way, however: I would much rather be linking a upnp library written in rust.
< sipa>
but i have no interest in seeing bitcoin core becoming developed in a mix of languages
< jamesob>
sipa: +1
< meshcollider>
Agreed
< * cfields_>
banishes sipa from the testing framework :p
< wumpus>
it'd be somewhat confusing, though I'd personally be happy to move away from c++, I've really grown to dislike it
< gmaxwell>
testing framework being written in another language has a non-trivial cost. (not that I'd suggest otherwise!)
< cfields_>
(That was a joke, I realize it's not the same thing)
< jamesob>
cfields_: I'm working on a c++ rewrite as we speak
< sipa>
wumpus: i realize that; but bitcoin core is a c++ project with c++ reviewers
< wumpus>
sipa: yes, maybe it means I need to move to rust-bitcoin :-)
< cfields_>
sipa: i think that's a fair point. But nearly every one of those reviewers that I've poked has mentioned that they'd like to learn rust.
< wumpus>
anyhow, this is an experiment, I don't think merging it is even a question right now
< MarcoFalke>
A lot of the reviewers know rust or wouldn't have a problem learning it
< gmaxwell>
It appears to me that language proliferation is doing a lot of harm to open source communties.... lots of duplication of effort and half abandoned projects just because someone wanted to do the same thing over in another language. :(
< sipa>
cfields_: sure, i'd like to learn rust; but i'm not going to be an expert in it even when i do to the extent that i'd feel like reviewing production ready code
< cfields_>
wumpus: right. So the question for now is how to get eyes on it and keep it up to date. I guess we can just maintain it in a branch somewhere?
< wumpus>
cfields_: yes
< meshcollider>
Review quality would definitely go down if people are new to the language
< luke-jr>
gmaxwell: if only everything used the same ABI
< cfields_>
sipa: very fair point.
< jeremyrubin>
Expertise in rust is easier to attain than c++
< jeremyrubin>
The compiler catches most of the stuff you spend time revieiwing in c++ anyways
< wumpus>
cfields_: as said I'm happy to maintain that branch
< gmaxwell>
I am skeptical also about this motivation about structurally elimiating bugs, when development in bitcoin core continues to _introduce_ bug in the form of things like memory-unbounded asynchronous layers.
< cfields_>
wumpus: ok, great, you're welcome to take it over.
< wumpus>
gmaxwell: concurrent programming should be more straightforward in rust at least
< gmaxwell>
wumpus: somewhat, but the problems we have like invisible queues that grow unboundedly exist exactly the same in rust.
< wumpus>
a lot of the boilerplate we're introducing in bitcoin for queues, asyn handling,et c is simply part of rust already
< wumpus>
gmaxwell: yes, no one is claiming it would eliminate all problems automatically, that would have been great
< gmaxwell>
It's a lot easier to avoid writing data races, however, indeed
< gmaxwell>
wumpus: not all problems, of course... but the problems we actually end up shipping.
< wumpus>
right
< wumpus>
I really don't know, like if you and sipa are 100% against including rust in bitcoin core, it's a done deal I guess
< sipa>
i'm not; but it's a discussion to be had when there is code to include
< wumpus>
yes
< gmaxwell>
As I said before, I'd be much happier with a rust miniupnp than miniupnp.
< jonasschnelli>
Let it be an experiment
< wumpus>
so let's start with miniupnp
< gmaxwell>
So certantly not 100% against it.
< jeremyrubin>
I think the point of this PR is that if someone wanted to start exploring, they would have to spend a week just setting up building and linking
< wumpus>
happy to write that in rust :)
< dongcarl>
I can get boostrap working
< sipa>
i'd be even happier with a minimal c++ reimplementation of miniupnp :p
< sipa>
(but i have no problem with a rust one if it would exist)
< wumpus>
you really want to make another c++ upnp implementation?
< sipa>
and it's great that the build system issues are already out of the way
< jeremyrubin>
So now that we have the demo build, someone can more easily show us motivation
< jeremyrubin>
It's also good to know that rust work won't be shot down purely on the "another language" basis
< gmaxwell>
I dunno, I think I'd rather a rust one, ignoring the build related issues (like rust notes, the blind binary trust in rust toolchains, etc)
< wumpus>
I'd not be inclined to trust that tbh, even if we wrote it ourselves, we're not perfect either it's not like we'd avoid all the problems in the original one automatically
< gmaxwell>
jeremyrubin: has the proble of the rust ecosystem where cargo effectively forces you into getting nearly blind updates been resolved?
< sipa>
i think we have a pretty good track record wrt the kind of bugs that miniupnp has
< wumpus>
you mean, xml parsing?
< sipa>
memory safety
< wumpus>
nah
< gmaxwell>
jeremyrubin: my expirence at blockstream was that it was very costly to not blindly take new software via cargo... because, yes you can ping versions, but then compiler updates would break deep dependencies, and you'd have to move forward your pins... and often move all of them at once because of interactions.
< gmaxwell>
s/ping/pin/
< jeremyrubin>
gmaxwell: I'm not sure? Crates.io is supposed to be append only
< sipa>
wumpus: how so?
< jeremyrubin>
gmaxwell: cargo also just added the ability to have your own crate registry, which would help
< wumpus>
sipa: just having a good track record doesn't guarantee anything
< sipa>
wumpus: of course
< jeremyrubin>
gmaxwell: in the PR, I manually pinned the one dep (a build tool to auto-gen headers)
< wumpus>
it doesn't mean you can do arbitrarily complex things and avoid bugs
< gmaxwell>
jeremyrubin: append only doesn't help if it doesn't build anymore with new compilers. (to be fair C++ also has this issue but on a 10x slower timescale)
< jeremyrubin>
gmaxwell: I think what grin does is not update the compiler -- just fix the version to a known stable or nightly
< luke-jr>
is it even possible to have multiple versions of Rustc installed on most distros?
< gmaxwell>
jeremyrubin: yes, thats what we ended up doing at blockstream... but then that runs into issues when you want to add something else and it depends on a new compiler.
< jeremyrubin>
gmaxwell: updating to a new compiler can be annoying for anything
< jeremyrubin>
luke-jr: yes look at the rustup toolchain manager
< gmaxwell>
luke-jr: the tooling makes it possible to do that, yes.
< sipa>
wumpus: to be clear, i say i'd prefer a c++ miniupnp because i'd be able to look at it; not because of innate qualities of the language
< wumpus>
I'm just very skeptical of things like 'if we implemented it it'd be better', I'm sure the author of miniupnp thought the same, he also wanted to make a minimal upnp implementation
< jeremyrubin>
gmaxwell: yeah -- it's a pity rust is adding so many exciting new features ;)
< wumpus>
it's pretty much outside all our expertise
< sipa>
but the whole question is pointless without an actual package to discuss
< wumpus>
yes
< wumpus>
(this is besides the rust issue btw)
< cfields>
Sorry, was laggy, had to switch servers.
< gmaxwell>
So there are essentially two infrastructure issues for using rust stuff for non-optional functionality: (1) bootstraping needing blobs, and (2) ecosystem very strongly pressuring everyone to take blind software updates (ala JS train wreak).
< wumpus>
I think using it for non-optional functionality is a bad idea right now
< wumpus>
agree on gmaxwell's comments re: cargo
< cfields>
gmaxwell: #1 is unfair and not worth discussing imo unless people are going to scream about us using ubuntu's toolchain as well.
< gmaxwell>
for something like a miniupnp, the main argument against using rust is that we might get fewer reviewers (but might get more rust fad-followers to review? so maybe thats a tie), the main argument I see for it, is that would be nice that there is a class of bugs that miniupnp had many of structurally prevented by the langauge.
< luke-jr>
cfields: people who build their own bitcoind *don't* need to use Ubuntu's toolchain
< gmaxwell>
cfields: we do use a toolchain that can be bit identically produced by many hetrogenious toolchains.
< jeremyrubin>
gmaxwell: for 2, there are a lot of things that could be done w/o external deps
< luke-jr>
gmaxwell: UPnP might be generic enough that non-Bitcoin devs are interested too
< gmaxwell>
cfields: in particular we use a gcc binary that I can produce starting from clang. (and I've independanty produced the compiler on my laptop from a disjoint toolchain)
< gmaxwell>
luke-jr: exactly.
< * dongcarl>
agrees, but needs help
< gmaxwell>
It's exactly the sort of thing likely to get somewhat broad interest, thats part of why I used it as a positive example.
< cfields>
Grr, I swore I wasn't going to take the bait on this :p
< wumpus>
apparently there are already some projects to implement upnp in rust
< gmaxwell>
cfields: I yield, I don't really need to argue this with you right now.
< wumpus>
in any case -- let's move to other topics, this isn't so urgent now
< jnewbery>
Can I quickly get back to #15750?
< gmaxwell>
cfields: we can agree that there are at least concerns there, even if we disagree how much its really any different.
< gribble>
https://github.com/bitcoin/bitcoin/issues/15750 | [rpc] Remove the addresses field from the getaddressinfo return object by jnewbery · Pull Request #15750 · bitcoin/bitcoin · GitHub
< wumpus>
#topic Remove addresses field in getaddressinfo
< jnewbery>
The addresses field was marked deprecated in v0.17 along with other changes to validateaddress and getaddressinfo (#10583)
< jnewbery>
#12490 (V0.18) should have removed the all deprecated functionality (and depracation notes). This part was missed out.
< gribble>
https://github.com/bitcoin/bitcoin/issues/12490 | [Wallet] [RPC] Remove deprecated wallet rpc features from bitcoin_server by jnewbery · Pull Request #12490 · bitcoin/bitcoin · GitHub
< cfields>
gmaxwell: oh for sure, this really is just "here, this is possible, see if it's useful". It seems it was taken as more than that.
< jnewbery>
So #15750 should fully remove it in v0.18.
< gribble>
https://github.com/bitcoin/bitcoin/issues/15750 | [rpc] Remove the addresses field from the getaddressinfo return object by jnewbery · Pull Request #15750 · bitcoin/bitcoin · GitHub
< gmaxwell>
cfields: +1000
< gmaxwell>
cfields: thanks for bringing it up
< jnewbery>
I don't think it counts as controversial if some random drops in and says "NEVER CHANGE RPCS". If we let that stop useful changes to Bitcoin Core we'd never do anything.
< wumpus>
jnewbery: I agree
< gmaxwell>
cfields: also I hope it was useful to the people most interested in using rust to understand the kind of headwinds that are out there. I don't think we can really have much useful discussion without talking about something specific.
< sipa>
jnewbery: his comments also seem to be about the move from validateaddress to getaddressinfo in the first place
< sipa>
which if anything are far too late
< luke-jr>
jnewbery: "bugfixes only during RCs" is not "NEVER CHANGE RPCS"
< gmaxwell>
It's unclear to me if he just needs help phrasing his argument.
< gmaxwell>
But I agree the argument as stated should be ignored.
< wumpus>
jnewbery: I was just reasoning 'what would it hurt to move this to 0.19', not trying to argue against the change!
< jnewbery>
Right, I don't think he's concerned about this particular field
< wumpus>
luke-jr: right
< instagibbs>
0.17 breakages were heavier than in the past fwiw
< gmaxwell>
Sometimes people have a real complaint (like "I use that field, can't easily generate it myself, and you're taking it away") but just don't know how to state it, so we should be sensitive to that.
< instagibbs>
not complaining, we just have delayed upgrading on some stuff due to it, which is fine
< jnewbery>
The problem is that the deprecation notice goes away in v0.18
< wumpus>
gmaxwell: yes, would make sense to add documentation for that
< gmaxwell>
(life would be better if bug reporters were assigned a lawyer)
< sipa>
gmaxwell: i'm... not sure
< wumpus>
(which is also what I noted in the PR)
< instagibbs>
are we suing people when they file issues :)
< MarcoFalke>
no, they sue us
< gmaxwell>
I just mean an advocate who would tell them how to state their issue in a way that makes sense to the process they're addressing.
< sipa>
ah yes
< jnewbery>
my point is that we should remove all of the deprecated parts of `deprecatedrpc=validateaddress` in one release or none at all
< luke-jr>
we can always say "use 0.17 until you're ready" too
< achow101>
jnewbery: ack
< jeremyrubin>
instagibbs: is the issue csw not being satoshi
< sipa>
jnewbery: i agree; but i also think it's a minor enough thing
< jnewbery>
Yeah, I wouldn't bring it up if we weren't already doing another rc
< wumpus>
yes, I don't care deeply either way, it just seems to cause a lot of discussion
< gmaxwell>
in any case, we should also consider it a good sign that we're getting compat complaints now. None is the wrong number.
< wumpus>
anyhow added "needs backport" for now
< jnewbery>
wumpus: thanks!
< gmaxwell>
sipa: thanks, will do
< sipa>
achow101: in particular, i think the new psbt descriptions could use slightly more text
< wumpus>
sipa: good idea
< sipa>
*new psbt rpcs
< achow101>
sipa: I can take a look at those
< wumpus>
yes, as a general concept, documentation belongs in the doc/ folder, not in the release notes, could always add a link :)
< sipa>
so also have a look at the updates in doc/psbt.md :p
< wumpus>
it's good to meantion things in the release notes but indeed not to be too wordy
< wumpus>
ok, cfields had another topic
< wumpus>
#topic Platform deprecation (cfields)
< wumpus>
sipa: will do
< cfields>
We don't (afaik?) have any means of deprecating old platforms. Many of the TODOs left for rust integration revolve around pretty old/unused platforms. I'm not suggesting that we drop platforms just because rust may be integrated at some point in the future, only that now seems like a reasonable time to re-evaluate. I originally planned to do the work to hack together fixes either here or upstream, but it's worth asking first: are these
< cfields>
platforms worth the effort?
< cfields>
The actual issues are documented in the PR. In particular, the troubled platforms are 32bit Windows and glibc < 2.15. glibc distro versions (unverified google hit) can be seen here: https://gist.github.com/wagenet/35adca1a032cec2999d47b6c40aa45b1
< wumpus>
which platform?
< sipa>
are all newly sold x86 CPUs these days 64-bit?
< wumpus>
yess since 2005 or so?
< sipa>
and do people run 64-bit OSes on them?
< cfields>
sipa: I would think anything running windows especially.
< wumpus>
for windows: yes
< sipa>
(i have no clue about the windows ecosystem in this regard)
< gmaxwell>
yes, though all still also capable of running in 32bit mode.
< luke-jr>
cfields: AFAIK current policy is to freely drop support for all but the most recent stable release of major distros; I assume none of them use glibc <2.15 ?
< gmaxwell>
for a long time people were still installing new systems as 32-bit but I think thats finally stopped as of about two years ago.
< cfields>
luke-jr: see link above, I don't recall off the top of my head.
< gmaxwell>
(e.g. two-ish years ago the default fedora download was 32-bit, in part because of compatiblity with #$@# binary only browser extensions)
< wumpus>
100% behind deprecating windows 32-bit
< jonasschnelli>
me 2
< gmaxwell>
unfortunately we don't run that well on 32-bit in any case, I'm not currently aware of a reason we shouldn't depricate windows 32-bit but I am not a windows expert. :)
< wumpus>
I've tried to raise the issue two years ago (or so?) and maybe one person complained on twitter, who was in the process of migrating things to a 64-bit OS later that year so…
< jonasschnelli>
just don't deprecate the ARM 32bits
< meshcollider>
Does that clean up any existing windows-only code or just for later
< wumpus>
and for 0.19
< achow101>
Windows does have 32 bit version still
< wumpus>
meshcollider: I don't think it cleans up any code tbh
< cfields>
Great, that was easy :)
< wumpus>
it just leaves oneless configuration to maintain and test
< wumpus>
which is good
< jeremyrubin>
Is anyone even testing 32-bit windows?
< wumpus>
no.
< jonasschnelli>
I do
< cfields>
wumpus: it's also the one that takes _forever_ to run.
< wumpus>
oh!
< jonasschnelli>
(VM though)
< luke-jr>
maybe we should solicit feedback somehow?
< gmaxwell>
12:52:42 < wumpus> I've tried to raise the issue two years ago (or so?) and maybe one person complained on twitter, who was in the process of
< gmaxwell>
^ fantastic!
< luke-jr>
open an issue to deprecate Win32, and put it in release notes?
< gmaxwell>
luke-jr: I don't disagree but it also sounds like wumpus already did previously.
< wumpus>
I mean we already dropped xp and vista support, which are the most likely to be 32 bit
< luke-jr>
gmaxwell: a tweet isn't as loud as release notes
< sipa>
luke-jr: i suspect more people read tweets :p
< luke-jr>
sipa: could be :/
< sipa>
(but maybe less relevant ones)
< cfields>
sipa: haha
< sipa>
windows 10 still has a 32-bit version it seems
< wumpus>
windows 7/8 was already mostly 64-bit, windows 10 certainly
< wumpus>
yes it exists but no one uses it
< luke-jr>
sipa: how many people use it?
< sipa>
okay.
< sipa>
luke-jr: i have no clue!
< bitcoin-git>
[bitcoin] jnewbery opened pull request #15800: [rpc] Remove the addresses field from the getaddressinfo return object (0.18...2019_04_remove_address_from_getaddressinfo_0.18) https://github.com/bitcoin/bitcoin/pull/15800
< luke-jr>
I'm sure Microsoft does :P
< luke-jr>
maybe we could just neglect to post the Win32 binaries for 0.18 and see who complains?\
< wumpus>
the kind of CPUs that only run 32-bit don't really run bitcoin, you'd be better off with a rpi at that point
< wumpus>
+x86
< luke-jr>
that way we can always upload them later if it's a problem
< meshcollider>
Seems sensible
< wumpus>
luke-jr: good idea
< wumpus>
:D
< gmaxwell>
12:57:31 < luke-jr> maybe we could just neglect to post the Win32 binaries for 0.18 and see who complains?\
< gmaxwell>
^ I'd +1 that too
< instagibbs>
wumpus, fwiw they croak when trying to run secp-zkp benchmarks too
< gmaxwell>
that would be a nice way of measuring usage.
< wumpus>
yes, let's do that
< wumpus>
instagibbs: whoops
< harding>
I'm not sure if people are serious, but if you are, please let me know in advance. The BitcoinCore.org site tests check for missing binaries.
< * luke-jr>
glares at the clock
< gmaxwell>
if people complain about the binaries, then we can just go ahead and post them.
< instagibbs>
harding, is that a complaint ;)
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Apr 11 19:59:36 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< luke-jr>
sipa: still here? can you post on --disable-asm issue?
< gmaxwell>
harding: We don't believe 32-bit windows is getting any substantial usage, we have no real way to tell... one way to tell would be to just not post binaries until someone complaints.
< luke-jr>
maybe we can rename it --disable-code-optimisations if you really don't like calling it asm
< sipa>
luke-jr: or split it up
< gmaxwell>
luke-jr: --boring-compiler
< luke-jr>
gmaxwell: that sounds like autotools mess
< gmaxwell>
There are two kinds of motivations that trigger wanting to disable asm: one is you're using an old/weird/limited/customized toolchain that can't handle them...
< harding>
gmaxwell: right, I understand what's being discussed. It just seems a bit weird to me. I don't actually care though, except that wumpus doesn't have to debug a travis failure on BitcoinCore.org at the last minute when he's trying to release 0.18.
< sipa>
luke-jr: i think i don't really care enough to argue :)
< gmaxwell>
the other is you're worried someone messed up the handcoded asm and introduced bugs
< luke-jr>
gmaxwell: another might be if you have some very interesting compiler you don't want asm to bypass?
< luke-jr>
sipa: well, apparently your last comment is being treated as a blocker :P
< MarcoFalke>
Looks like a rename will please sipa
< sipa>
wumpus: btw, i shouldn't have brought up the rust vs c++ in context of miniupnp; i'm skeptical about a push for rust code, but if there was a upnp replacement in rust with reasonable assurances about production readiness, i'd be all for it
< luke-jr>
release notes: I don't understand the change described in "Deprecated P2P messages"
< BlueMatt>
do we have download counts/should we monitor download counts?
< luke-jr>
BlueMatt: I have been fetching download counts from the PPA for a few releases now
< BlueMatt>
that doesnt help much for 32-bit windows :p
< luke-jr>
ah
< wumpus>
so I"m removing two files from the upload: bitcoin-${VERSIONTO}-win32.zip and bitcoin-${VERSIONTO}-win32-setup.exe
< wumpus>
@ harding
< luke-jr>
wumpus: well, I expect we should still build/sign them, just not upload
< wumpus>
luke-jr: that's what I'm saing
< luke-jr>
k
< wumpus>
not changing the gitian descriptor
< wumpus>
can do that for 0.19
< luke-jr>
can I put FIXME notes in the relnote wiki?
< harding>
wumpus: received. Thanks; I've already patched the test and will include it with the ready-for-release PR I'm opening today.
< luke-jr>
or is there a better place for that?
< wumpus>
harding: thanks!
< wumpus>
luke-jr: only if you intend to make the fixes yourself before final, I guess
< luke-jr>
wumpus: I'd just make the fixes, if I knew what they should be
< luke-jr>
wumpus: the problem is I have no idea what changed :P
< luke-jr>
(which IMO indicates the release note is insufficient in describing the change)
< wumpus>
would be kind of awkaard to accidentally get people's fixme's and random notes *for the release notes* into the release notes
< harding>
luke-jr: do you think it should be moved to the Planned Changes section?
< wumpus>
sipa: makes sense
< luke-jr>
harding: if that's all it is.
< harding>
luke-jr: I didn't write that note, but I'm guessing it's there because we planned to disable it by default in this release and then backpeddled to planning to disable it by default next release.
< luke-jr>
hm, that would make sense
< luke-jr>
yeah, I'm not seeing anything else in the git log
< harding>
I think the previous discussion was that we don't actually have any way to indicate that a P2P message is deprecated besides creating a release note.
< luke-jr>
heh, there's already a TODO in there
< wumpus>
a BIP would be the expected way I think
< luke-jr>
dunno, this is just Core deciding not to support a BIP
< gmaxwell>
We often decide to not support bips, less often decide to remove bips.
< wumpus>
but I mean, theoretically a BIP could describe the reasons for not supporting a BIP anymore, for reconsidering its use
< luke-jr>
gmaxwell: sure, but it's still an application-specific thing
< sipa>
wumpus: i guess that would make sense if we believe there is a reason why others should also not support something
< wumpus>
sipa: yes
< sipa>
it's a proposal for improvement; if not supporting cam be argued to be an improvement, it deserves being a bip
< gmaxwell>
Right, someone might want to make a case for "no one should support this"
< wumpus>
so things like 'bandwidth overhead'
< wumpus>
or potential privacy leaks
< luke-jr>
gmaxwell: seems like a case for BIP Comments ;)
< gmaxwell>
though some of the arguments are just structural, like in general returning normative or somewhat normative error codes imposes significant architecural obligations.
< gmaxwell>
As a general principle, tests should be able to test all visible network behavior.... and refactoring your code shouldn't break tests ... returning errors can make otherwise clean abstractions unclean.
< cfields>
meshcollider: Were you going to look at getting rid of some of the win32 stuff?
< cfields>
(if any)
< wumpus>
I'm surprised qt still supports win32
< wumpus>
though ok, qt is commonly used on embedded hardware so also not that surprised
< luke-jr>
portability is (was?) kind of Qt's main selling point
< cfields>
Yeah, I bet there's still a ton of embedded win32 stuff out there.
< cfields>
I used embedded winxp in some microscope equipment at one point, that was wacky.
< luke-jr>
but Qt did drop WinXP right?
< cfields>
meshcollider: If you do take a look, I'd request that we maintain the multi-windows machinery and not just squash down to "windows" as a platform. That way we don't have to start all over if they pivot to some new arch.
< cfields>
Basically, I'd suggest not doing something like s/#ifdef win64/#ifdef windows/.
< luke-jr>
yeah, since ppc64 is taking over /s
< cfields>
Heh
< cfields>
I was thinking more like their 10th attempt at an arm platform.
< wumpus>
windows rv64 would be a cursed combo
< wumpus>
so I think it'll happen at some point
< cfields>
lol
< luke-jr>
as if RISC-V isn't wide open to being closed down
< cfields>
gotta gotta get up to get down.
< gmaxwell>
More likely to see google or oracle produce a embrace-and-extent RV than msft these days. :P
< cfields>
gmaxwell: fuchsia seems poised to help with that.
< gmaxwell>
oh is someone actually doing riscv-cheri now?!!?!?!
< gmaxwell>
I was bummed when riscv didn't include cheri like features from day one, though I understood why...
< wumpus>
right, would have been too complicated and experimental I think, but it might be part of the rationale behind RV128
< gmaxwell>
ISTM its really hard to really compete with intel/amd x86_64 duooply for raw performance... basically only ppc does. But it would be easy to smoke x86_64 on security, and I think there are a lot of applications where even the in-order arms are fast enough, but more security would be desirable.
< wumpus>
right, one exciting thing about RISC-V is that it allows for experimentation like that, it's no longer restricted to what AMD and Intel or ARM think as feasible commercially
< wumpus>
a lot of things don't really don't need that much performance, or, if they do, it's better served using some FPGA or custom ASIC
< wumpus>
also because it can give actual timing guarantees
< anddam>
has issue #1390 ever been discussed again since 2012?
< wumpus>
anyhow I hope CHERI will see more adoption, it's a lot more thorough approach than trying to manually sandbox every single thing
< wumpus>
anddam: no, I don't think so, though some options have been added since to put different things in different dirs
< anddam>
like --datadir, --wallet and the config file one?
< wumpus>
yes and -blocksdir
< wumpus>
-walletdir
< anddam>
have there been rewrites of bitcoin-core in other languages?
< wumpus>
it's very flexible where to put things
< anddam>
wumpus: option in a config is what I ended doing, along with the small desktop file change to pass --datadir, I do not like program that write dotfiles in home, ls -a quickly becomes a mess
< wumpus>
it can also be useful because of backups, put the blocks on a large more-or-less throw-away partition, but the wallets in a secure place
< anddam>
wumpus: yep
< anddam>
wumpus: when starting bitcoin-qt does a separate bitcoind process get spawnd?
< anddam>
(I cannot start bitcoin-qt right now to check)
< anddam>
I wonder how detached the Qt GUI and bitcoin core are