< jonasschnelli>
gitian: is there a solution if make-base-vm complains with "E: Couldn't find these debs: git-core"?
< jonasschnelli>
Is that an Apt-Cacher-NG issue?
< Fuzzbawls>
jonasschnelli: AFAIK all references to "git-core" have been replaced with just "git" in the gitian descriptors. could be a local caching issue (though I myself have never encountered such a thing).
< ken2812221>
jonasschnelli: Use the latest version of gitian-builder
< jonasschnelli>
Thanks... will try
< jonasschnelli>
ken2812221: updating gitian-builder fixed the issue. Thanks
< jonasschnelli>
I didn't updated since I'm pretty sure I added some local modifications. :)
< Fuzzbawls>
did you ever get a self-compile of LXC 3 working on debian? think i saw it was you that was trying to use version 3...or maybe a later version 2 that wasn't supplied by the distro packages
< ken2812221>
IIRC make-base-vm does not use apt so it may not know about package alias.
< jonasschnelli>
Fuzzbawls. I self compiled 2.1.1
< provoostenator>
jonasschnelli: got it, thanks
< ken2812221>
bitcoin-git is dead?
< provoostenator>
jonasschnelli: the Windows build thinks it's version b591ece04 rather than 5ca74904
< achow101>
ken2812221: sipa killed it by setting +n
< achow101>
to prevent spamming that was happening
< provoostenator>
I guess it makes a merge commit first
< ken2812221>
achow101: thanks
< jonasschnelli>
ken2812221: any idea how to fix "E: Package 'curl' has no installation candidate" (inside the gitian VM) during gbuild?
< gmaxwell>
BlueMatt: it's called the redeemScript.
< gmaxwell>
Or witnessScript.
< BlueMatt>
gmaxwell: hmm witnessScript is confusing af, imo
< arubi>
redeemscript is even more confusing when it's say a p2sh-p2wsh transaction
< achow101>
it's been called the witnessScript. Changing it now would probably introduce more confusion
< arubi>
DDG first result for "scriptWitness" is the transaction.h file in the repo, and for "witnessScript bitcoin" the first result is is the core-dev docs site
< arubi>
+1 witnessScript
< arubi>
(posted on the issue)
< BlueMatt>
arubi: well you could call it "witness redeem script" pretty easily
< BlueMatt>
scriptWitness already refers to the full witness
< BlueMatt>
so now scriptWitness and witnessScript are different things?
< arubi>
"witness redeem script" might be better than "witness redeemScript" if it's going to be called that then. and yea I see your point about this but at least these two terms are easily distinguishable in search
< BlueMatt>
from my github comment: "Also, further confusing is that its easy to see the witness as a replacement for the scriptSig (though that's not entirely accurate due to it being a list of pushes, not an executed script), at which point scriptWitness/witnessScript would be easy to assume referred to the full witness."
< BlueMatt>
funny that people had been calling it witnessScript and I'd never actually seen that anywhere lol
< arubi>
maybe "witnessSource" ? sort of the source code for the witness program? :)
< BlueMatt>
I mean I dont hugely care, I just think witnessscript/scriptwitness is absolutely a terrible idea
<@sipa>
awww i'm sorry :)
< sipa>
witnessscript = script in the witmess
< sipa>
scriptwitness = witne for a script
< BlueMatt>
ok, so given there's already like three terms to describe witnessscript, lets stop calling it witnessscript :p
< sipa>
witness redeemscript sgtm
< satwo>
Hi all. BIP-141 defines 4 ways to measure the size of a transaction: weight, virtual size, base size, and total size. Bitcoin-cli decoderawtransaction returns weight, vsize ("virtual size" - obvious), and size (“total size" - not obvious). I must not be the only one to have found it nontrivial to figure out how base size, total size in BIP141 and “size” in RPC are related. Even once one figures out that “BIP 141
< satwo>
total size” = “RPC size”, base size and witness data size must be calculated with a little backwards-engineered DIY algebra. Is there room for improvement in documentation/RPC fields here, or am I missing something?
< BlueMatt>
sipa: plz2comment on bitcoincore.org issue, then
< gmaxwell>
satwo: "base size" is basically completely meaningless in the protocol. It's not used for anything.
< sipa>
BlueMatt: what issue?
< gmaxwell>
(okay, it's used in the minimum size standardness rule, but I think nowhere else)
< satwo>
gmaxwell: That was my intuition, but some things threw me off; i.e. its being used as a tx field in BlockSci, and the fact that many block explorers seem to refer to the base size of a tx in their size field (other explorers refer to total size)
< sipa>
satwo: explorers should only show vsize
< sipa>
all the rest are technical details that most users won't care about
< gmaxwell>
satwo: unfortunately people have promoted a lot of crazy misunderstandings that make people think things that matter don't.
< gmaxwell>
e.g. people saying that the limit is "1mb base + 3mb witness", which is not at all how it works, but if it did it would make sense to print two size figures.
< satwo>
sipa: very few do, it seems. For example the tx 88d87642bc1534b9d6f8d62e6e9ae55e5971c0efec30d9139f817eb55c307c71 has a "size" of 381 on Blockchair, blockchain.info, btc.com, and smartbit.au, and a "size" of 126 on Blockcypher, Blocktrail and blockexplorer.com. 381 is the total size and 126 is the base size. So clearly there's some confusion with nomenclature
< satwo>
Of course vsize is nowhere to be found except on smartbit
< sipa>
satwo: i'm aware
< sipa>
i've contacted a few
< sipa>
it was probably a mistake to introduce a new namw.for it; we should jusr have replaced size everywhere with vsize
< sipa>
but that would have run into issues with people who assumed size = len(serialization)
< satwo>
Would modifying bitcoin-rpc to say something like "size (total):" or "total size:" be messy overkill? At the very least it would bring RPC and BIP 141 in harmony, potentially reducing some confusion
< gmaxwell>
We should probably drop size out of the rpcs.
< BlueMatt>
I have a somewhat strange topic: what to call the witness version of the p2sh redeemScript...not quite the right venue to discuss it, but there's not a much better one and we have to pick something for bitcoincore.org, sooooo
< wumpus>
#topic naming of witness version of the p2sh redeemScript
< achow101>
hasn't this been called "witnessScript" for a while?
< sipa>
yes
< achow101>
that's what i have used for bip 174 at least
< BlueMatt>
I had never seen that
< BlueMatt>
though I admit it was in the BIP
< BlueMatt>
and I know people who've called it the witness redeem script or so
< BlueMatt>
which is also confusing cause of p2sh-wrapped segwit
< BlueMatt>
but witnessScript is confusing given scriptWitness refers to the whole witness :(
< jonasschnelli>
how is this important to define?
< BlueMatt>
so every option is shit
< sipa>
perhaps it should be called P2WSH redeemscript, as it's arguably specific to P2WSH (P2WPKH doesn't have it, and future witness versions may not either)
< BlueMatt>
jonasschnelli: well we need to call it *something* and it seems everyone has a different one
< kanzure>
using ambiguous jargon will cause errors and bugs
< sipa>
BlueMatt: scriptWitness is just in bitcoin core's source code though; is it called that way anywhere else?
< BlueMatt>
sipa: I'm not sure that it is, but that was MarcoFalke's comment to me
< jonasschnelli>
IMO it's specified in the BIP, but people are free to form a new term. I don't think there is need to an authoriity to define it.
< BlueMatt>
but, given the witness can be seen as a "scriptSig replacement" calling it that I could see being incredibly confusing to some people
< sipa>
yes, i agree it's confusing and we could have picked a better name
< BlueMatt>
jonasschnelli: well I ask because there is debate about what to write in some docs in rust-bitcoin, and also what to call it on bitcoincore.org docs
< sipa>
the cat may also already be out of the bag since 2 years ago
< BlueMatt>
jonasschnelli: so this is the right venue to discuss bitcoincore.org
< BlueMatt>
sipa: sure, but I've seen it referred to as other things too already :(
< promag>
hi
< gmaxwell>
I think this discussion is a waste of time for this venue.
< provoostenator>
For now, maybe just explain that it's confusing and someone should propose a BIP to deconfuse it?
< kanzure>
cfields: are the poll results due today?
< jonasschnelli>
Can we also rename "wallet"? *duck*
< cfields>
kanzure: ah, thanks for the reminder. poll closed at the end of yesterday's meeting. winner: current time
< cfields>
er, last week's meeting
< wumpus>
#topic meeting time
< provoostenator>
Even just pointing out that something_is_ confusing, helps the reader pay attention, otherwise they might think they just don't get it.
< sipa>
also w.r.t. scantxoutset, are we going to mark it experimental?
< wumpus>
I think everyone agreed on that
< jonasschnelli>
Yes. I can PR that.
< promag>
+1
< gmaxwell>
jonasschnelli: thanks.
< jonasschnelli>
Sorry for the delayed review on sipas descriptor work... will comment soon on the PR
< wumpus>
#topic 0.16.2 final
< BlueMatt>
ack
< wumpus>
rc2 was tagged ~a week ago, I don't think any issues came up
< gmaxwell>
I haven't seen or heard any issues with the RC.
< wumpus>
so I think it's time to tag final
< jonasschnelli>
agree
< cfields>
+1
< wumpus>
ok, will do so after the meeting
< gmaxwell>
not have any OMG-must-fix-now bugs cropped up that I'm aware of.
< promag>
+1
< achow101>
yay
< wumpus>
any other topics?
< cfields>
quick personal announcement: A small health issue has been taking up a good amount of my time lately, and I've been struggling to keep up with review, let alone writing new code. I've decided to take a week or two to try to finish up outstanding things, then take a month away to try to get back to 100%. I'll try to at least keep up with emails and pings during that time.
< ken2812221>
Is it allowable to add wmain function?
< wumpus>
#topic encoding issue on windows (ken2812221(
< cfields>
there are a bunch of current PRs for depends and gitian descriptors. I assume it's no problem to continue working on those for 0.17? There are a few fixes that may be non-trivial that I would greatly prefer over the one-liner fixes.
< wumpus>
ken2812221: I'd prefer not, I think we had multiple entry points at some point, with special one for windows but this was cleaned up to just main(), if there is really no other way
< gmaxwell>
i hate strings
< wumpus>
so do I, but unfortunately they're needed for path names
< cfields>
windows strings cause 2x developer hate :(
< luke-jr>
they string us along?
< gmaxwell>
so the issue here is that windows APIs want UTF16 strings or something?
< cfields>
luke-jr: i would characterize it that way, hes
< gmaxwell>
Originally it was UCS2 but then they realized that chinese exists and it became UTF16 to get the worst of all worlds or soemthing like that.
< wumpus>
is this reallky all necessary? it changes pretty much all uses of paths in the code
< sipa>
yeah, they adopted unicode very early, and picked a different encoding than what the rest of the world eventually ended up pickin
< gmaxwell>
ken2812221: what keeps you from intercepting a couple places at a low level and inserting at UTF8->UTF16 conversion?
< wumpus>
I hate waltzing over the entire code to accomodate windows' crappyness
< ken2812221>
The command line argument, I think.
< wumpus>
most of the changes seem .string() versus .u8string()
< gmaxwell>
ken2812221: the commandline arguments come in as utf8 strings, right?
< ken2812221>
no, it is ANSI or UTF-16.
< wumpus>
on POSIX platforms that's what we assume, in windows they come as utf16 strings
< ken2812221>
on Windows
< sipa>
since windows 10 apparently you can select a codepage for the "ansi" encoding that is utf8
< wumpus>
sipa: oh!
< sipa>
oh no
< gmaxwell>
wumpus: so how do we deal with like ua comment coming in and not sticking UTF16 into our network messages?
< sipa>
only possible since windows 10 insider build 17035 (November 2017)
< cfields>
gmaxwell: we sanitize those
< gmaxwell>
cfields: but given UTF16 wouldn't our sanitizer just corrupt the string? (throw out all characters?)
< cfields>
(unsure what gets lost in the conversion, but we know what can't go out)
< wumpus>
gmaxwell: it needs to be converted to UTF8 for the internal use
< gmaxwell>
okay, I clearly know nothing here so I should probably just go read.
< wumpus>
so the arguments come in as UTF-16, then are converted to UTF-8 for storage
< gmaxwell>
wumpus: right I guess I was just assuming the argument processing conerted it all to utf8 before we saw any of it.
< wumpus>
which makes complete sense in ken2812221 's PR
< cfields>
gmaxwell: nah, I think you're right. I was just making the point that it at least won't go over the wire that way.
< sipa>
ok, 17035 was finally released as "April 2018 update"
< sipa>
that's... a decade too late
< wumpus>
sipa: yes...
< gmaxwell>
In which case I'd assume the path issue could be solved by wrapping the file IO with something that converts our internal utf8 to utf16 for windows.
< luke-jr>
XD
< wumpus>
though microsoft is twisting people's arms really hard to upgrade to windows 10
< sipa>
don't we already have the fs space for that?
< wumpus>
yes, that's what his PR does
< sipa>
hmm, i would expect it's just changing one or two functions
< gmaxwell>
^
< sipa>
sorry, i'm not very familiar with this part of the code; i should probably go look
< wumpus>
it makes sense, the only thing is dislike is the size of the diff because he uses .u8string instead of .string in so many places, but it's fairly simple
< ken2812221>
There are some TODO: leveldb and fstream
< wumpus>
should probably get over it and review it...
< ken2812221>
They are not support utf-8 in this PR.
< wumpus>
so that will still fail with datadirs with, say, Chinese characters in it?
< ken2812221>
Yes, still fail, but success if you set your setting to Chinese.
< ken2812221>
Before this PR, both fail.
< wumpus>
okay, that's good
< ken2812221>
Thanks
< jtimon>
I guess #13311 doesn't deserve to be in the 0.17.0 milestone since it is not a feature nor a bugfix
< wumpus>
it's not really about 'deserve', I don't see a point to add it to the milestone, but if it is ready for merge before branching off 0.17 it can make it into 0.17
< jtimon>
sure, perhaps not the right word, "I guess there's no point for in to be in the milestone"
< wumpus>
yes, I agree with that
< jtimon>
I guess I was just review begging by mentioning it, sorry
< wumpus>
that's okay
< wumpus>
I guess we're out of topics
< wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Jul 26 19:56:16 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< gmaxwell>
Does anyone have any idea on how we could tell if dbcache flush events are becoming correlated across the network?
< wumpus>
no idea how to do that without adding instrumentation and metrics reporting
< gmaxwell>
I suppose we could check logs from a number of our own long running nodes and see if they line up, but I bet most of us are changing the default dbcache size.
< skeees>
thats a pretty tricky, but interesting problem, would flushing db cache introduce a measurable delay in responding to a ping?
< skeees>
how close of an interval becomes problematic in your opinion?
< wumpus>
tes but so does validation of blocks
< wumpus>
gmaxwell: indeed, logs would also be good
< gmaxwell>
skeees: yes but validating a block does too
< gmaxwell>
Basically the concern is that if nodes are synchronizing their flushes, it will slow down block propagation somewhat.
< gmaxwell>
Because you get a block, validate it, flush (if needed), aand relay it.
< gmaxwell>
if nodes aren't synced propagation will just route around flushing nodes.
< gmaxwell>
HB mode with forward-before-validate like we have now I'm sure greatly reduces the issue.
< kanzure>
relay and flush are in same thread?
< gmaxwell>
I was talking to matt in private about some relay improvements, and we observed that even for relay the stuff we were talking about was probably not the low hanging fruit.
< gmaxwell>
kanzure: non-oppturnistic realy blocks on validation (as it must per protocol), and validation blocks on flushing (as it kind-of must to avoid excess memory usage)
< skeees>
well i do have that pr that puts net and validation (/flush) in separate threads - it would still require additional work though to remove cs_main from relay I think
< gmaxwell>
in theory everything could be rejiggered to make that less bad, but that effort would probably be better spent in making flushing a non-issue via incremental flushing which we changed the design to facilitate a few releases ago.
< gmaxwell>
skeees: putting it in seperate threads is irrelevant.
< skeees>
that wouldn't enable you to relay while the flush happened?
< gmaxwell>
A question of threading isn't the source of delays.
< kanzure>
it's validation. if you want non-opportunistic relay.
< kanzure>
hence the mention of a refactor
< gmaxwell>
skeees: you'd be able to do just the same by moving the flush ahead of the next verification operation, without changing anything with threading.
< gmaxwell>
in any case, I asked because if they're syncing it's probably a completely safe one line of code change to make them no longer sync up.
< gmaxwell>
(just make each node randomly make its dbcache zero to four blocks of worth of data smaller.)
< wumpus>
* [new tag] v0.16.2 -> v0.16.2
< gmaxwell>
The right bigger change is to make it so that we only flush a small amount, and then flush every block. A few versions ago we relaxed the invarient that the chainstate database has to be consistent with a particular block.
< sipa>
wumpus: \o/
< gmaxwell>
But even that isn't worth doing for latency reasons; it's worth doing because it should speed up sync a lot by better overlapping writing.
< gmaxwell>
(not worth it for latency because esp with oppturnistic sends, latency is already stupid low)
< skeees>
if you flush every block - won't that affect propagation of blocks that come in very close succession? (i assumed this was why you suggested a randomized flush)
< gmaxwell>
skeees: if its flushing every block each flush will only take a fraction of a millisecond.
< gmaxwell>
right now a 'flush' writes out the entire dbcache, not just one block worth of it.
< skeees>
ah
< gmaxwell>
that used to be required so that the chainstate on disk would be consistent with a specific block, but we don't require that anymore.
< gmaxwell>
because on restart we'll just replay the effect recent blocks had on the chainstate.
< jtimon>
randomly reducing the cachedb a little bit seems like an inofensive temporal fix, also simple to remove when we have the better fix