< dgenr8>
adiabat: bloom filter digests would be great, but won't kill off connection bloom filters. people want to see unconfirmed activity
< gmaxwell>
dgenr8: the addition of filtering the current transactions was a 12th hour addition to BIP37, and for many users saving an average of 13kbit/sec for a total loss of privacy is not all that interesting just to see unconfirmed transactions that their wallets can't even tell are at all remotely possily correct.
< dgenr8>
gmaxwell: so you think wallet should be watching all live tx, or none, or "either of those, just not filtered"?
< gmaxwell>
Or filtered, it's the wallets call. The exact behavior depends on the specific use case, and I think the specific use case for filtered is fairly narrow.
< gmaxwell>
If the digests idea had come up at the time of BIP37, I think it would have been implemented and filtering of relayed transactions wouldn't have been.
< dgenr8>
one use case is "i'm told a tx was broadcast that pays me. i wonder if that's true"
< gmaxwell>
P2Pool is updated for CSV.
< luke-jr>
it needed updating?
< luke-jr>
oh, because peers need to police each other
< gmaxwell>
also because it compensates for lack of softfork flags on gbt by not signaling higher versions than p2p knows about.
< GitHub140>
[bitcoin] gmaxwell opened pull request #8179: Evict orphans which are included or precluded by accepted blocks. (master...reap_orphans) https://github.com/bitcoin/bitcoin/pull/8179
< luke-jr>
gmaxwell: ah, right
< iniana>
Will miners who don't upgrade get their blocks orphaned immediately once the grace period is over (by not signalling the activated bit) or only when they mine an invalid block?
< GitHub0>
bitcoin/master 77f63a4 Pieter Wuille: Fix two warnings for comparison between signed and unsigned
< GitHub0>
bitcoin/master 7e6dd7b Wladimir J. van der Laan: Merge #8172: Fix two warnings for comparison between signed and unsigned...
< GitHub101>
[bitcoin] laanwj closed pull request #8172: Fix two warnings for comparison between signed and unsigned (master...fixunsigned) https://github.com/bitcoin/bitcoin/pull/8172
< gmaxwell>
Hm. I think with the 0.13 sorted inv behavior can actually wipe out the orphan map when we have no outstanding unresponded getdata for transactions from any peers. (this wouldn't be a good idea with the pre 0.13 behavior in the network, so we probably shouldn't do that now)
< jonasschnelli>
We need a decentralized github web. :)
< sipa>
You mean git?
< jonasschnelli>
hehe...
< jonasschnelli>
Yes. With the option of creating issues, public announcements of pull requests, etc.
< wumpus>
there have been ideas to create a git.bitcoincore.org which mirrors the repository and where one can clone from if the github repository is offline, the thing is, no one really has time to set up and babysit such things
< warren>
Does the git repo use signed merge commits?
< btcdrak>
wumpus: maintenance is a bitch.
< sipa>
warren: we always gpg sign merge commits
< btcdrak>
warren: yes.
< sipa>
warren: and there is a script to verify that
< jonasschnelli>
wumpus: you mean mirroring the github data?
< jonasschnelli>
Or just the git?
< wumpus>
just the git
< jonasschnelli>
Okay. That would be trivial.
< wumpus>
we do mirror the github data in a github repository, could also push that there
< jonasschnelli>
btcdrak: where is bitcoincore.org hosted?
< btcdrak>
I'm sure the odd Github Unicorn doesnt cause us _that_ much trouble.
< btcdrak>
jonasschnelli: the website is at github :)
< sipa>
wumpus: wgat
< jonasschnelli>
Yes. But we can't work on a decentralized system on a centralized github. :)
< btcdrak>
but can set a DNS entry to any server...
< btcdrak>
(a subdomain I mean)
< btcdrak>
jonasschnelli: decentralisation is a myth in this case.
< wumpus>
we've been signing merge commits consistently from ~2013 or so, and were one of the first projects to do this
< sipa>
unfortunately git commit ids are only sha1 :(
< wumpus>
well it's not perfect security but certainly puts of a barrier, which is the point of any security feature
< btcdrak>
i love how this entire conversation comes up almost verbatim again every now and again :)
< wumpus>
we're like broken records
< btcdrak>
mayeb I should write a FAQ :-p
< btcdrak>
but seriously, my point is: someone sees a Unicorn, and rather than press F5, it ignites a sledge hammer to crack peanut response :-D
< wumpus>
please include a "database corrupted" entry
< wumpus>
really doesn't help that everyone creates a new issue every time they get that
< wumpus>
where 99 out of 100 times it's simply a hardware problem
< btcdrak>
agreed, I think there are a bunch of common issue we could cover.
< wumpus>
we really should discourage people from creating new issues about that, unless there's new information to report
< wumpus>
maybe they could post to an existing issue for statistics or something
< gmaxwell>
change the message to "Apparent hardware error" :)
< wumpus>
gcc has the same thing, where people keep reporting SEGV crashes caused by overclocked or otherwise flakey CPUs, while compiling run-of-the-mill software
< wumpus>
gmaxwell: yes, that would be a good idea
< paveljanik>
maybe even a link to the FAQ entry about this?
< sipa>
nobody believes you when you tell them their hardware is broken
< gmaxwell>
a few do.
< wumpus>
still, it helps to have it written down somewhere
< wumpus>
in a clear way
< wumpus>
instead of repeating it all the time, increasingly annoyed :)
< gmaxwell>
I actually think they believe it from the sofware more than names on IRC.
< gmaxwell>
well whats worse is that they search the log entry, and find other people explaining that it's totally not hardware and the software is buggy and what not'
< gmaxwell>
and they show up "hey you morons still haven't fixed this!"
< gmaxwell>
they also run -rescan ten times and it won't go away and why should they have to run it an eleventh!?
< gmaxwell>
Is there a german word for where you both feel apologetic towards someone and want to strangle them?
< wumpus>
the thing is, even if there are software bugs involved, opening more issues about it isn't going to help a thing
< gmaxwell>
A lot of people just don't have a good mental model of how this whole thing works. Like, if it's still buggy and reported, then why haven't we fixed it? obviously we don't care or are lazy.
< wumpus>
I think I'm going to create one 'data corruption' issue and close the others as duplicates
< gmaxwell>
Same reason why reports seldom have the information required to reproduce the issue.
< wumpus>
well most issues are fairly good in that regard
< gmaxwell>
if we had simply forgotten, reporting again would help-- and we do, from time to time, forget actual issues.
< wumpus>
generally not issues that are reported on github
< gmaxwell>
Right.
< wumpus>
sure, if something is said on IRC or the mailing list it gets lost
< gmaxwell>
at least not while they're still open.
< wumpus>
the point of an issue tracker is to have a persistent URL for an issue
< gmaxwell>
But clearly we've fogotten the data corruption because it keeps hapenning. :P QED.
< wumpus>
if it gets too crowded it loses its value
< wumpus>
then it's just like mentioning it once on reddit, no one will ever look back to it
< wumpus>
heh
< wumpus>
well every time I had actual data corruption I investigated in depth
< jonasschnelli>
What about providing a script/cli-tool (C++) that would perform some heave leveldb tests with block like data? Something like a bitcoin-hardware-test?
< gmaxwell>
it's called bitcoind.
< wumpus>
it will take long to detect errors, so no one will run that
< wumpus>
heh exactly
< gmaxwell>
it's not just leveldb though, the cpu load from signature validation triggers memory errors on some systems.
< jonasschnelli>
Okay.. :)
< gmaxwell>
when we have wumpus' snapshot stuff perhaps we could also do other things to retry/recover in the face of error.
< wumpus>
there are just so many types of hardware problems that can result in corruptions
< gmaxwell>
I have mixed feelings, users really shouldn't count on faulty computers... esp for handling irreversable money...
< wumpus>
and there is software to detect hardware problems, that's a completely orthogonal thing :)
< jonasschnelli>
Right. Agree.
< gmaxwell>
many are resolved just by being able to go back to earlier state... obviously flying writes that corrupt data at rest.. well not much then can be done there... (well, we do have forward error correction code...)
< jonasschnelli>
I just had serval crashes on my Pine64 (due to false configuration and maybe USB issues). I always had to reindex/IBD from the scratch.
< jonasschnelli>
Wumpuses snapshot idea would be a live-safer for such situations
< MarcoFalke>
wumpus: Can you explain why you left BUILD_DATE in share/genbuild.sh ?
< MarcoFalke>
Is it used elsewhere?
< wumpus>
MarcoFalke: probably an oversight, or I wasn't sure, let me see
< wumpus>
yes that can definitely go
< instagibbs>
is there a list of known issues with getbalance? I'm getting inconsistencies, and the comments in the code give me instructions which causes a json parsing error(??)
< instagibbs>
"// getbalance and "getbalance * 1 true" should return the same number"
< instagibbs>
oh the latter is probably me being dumb, but I still am getting weird balances
< dgenr8>
instagibbs: getbalance and getbalance "" appear to differ in their treatment of 0-conf utxos from self. getbalance "" 1 seems to always includes them. getbalance "*" follows from getbalance "" (since "" is just a specific account)
< dgenr8>
instagibbs: also sendfrom "A" seems to not work in a test I just just (it selected a utxo not belonging to "A")
< dgenr8>
justdid
< instagibbs>
yeah i opened an issue
< instagibbs>
since im not sure what the comment should be changed to
< instagibbs>
there are nooks and crannies depending on settings you are running
< dgenr8>
agreed that since -spendzeroconfchange affects the result of getbalance, the comment is too general. prolly just remove it
< sipa>
dgenr8: sendfrom just subtracts the balance from A; it always uses all utxos
< sipa>
dgenr8: as accounts don't own utxos in any way
< instagibbs>
dgenr8, yes I think removal is best
< sipa>
dgenr8: also, do you feel like addresses the comments on your partition check improvement?
< wumpus>
first at PSA: the feature freeze for 0.13 is next week. Make sure that whatever features need to be merged are merged before that time. If there are any pulls that require special attention, or are ready, let me know.
< cfields_>
wumpus: I have 2 p2p refactor PRs that i'd _very_ much like to have in 0.13. I'm not sure how you're considering those in terms of freezing
< MarcoFalke>
cfields_: I think p2p refactor can go in after the feature freeze?
< gmaxwell>
We apparently can no longer compile on hosts with only 2GB ram with defaults.
< wumpus>
other TODOs from last week: review and merge #8126 (std::shared_ptr based CTransaction storage in mempool) - that was done, #7935 (Versionbits: GBT support) - also done
< MarcoFalke>
I mean it is not a new feature ;)
< gmaxwell>
well it was more like 1.5GB ram before.
< wumpus>
others have not yet finished: #7598 (Refactor CreateNewBlock to be a method of the BlockAssembler class)
< jonasschnelli>
gmaxwell: I compiled on a 2GG AARCH this week successfully.
< jonasschnelli>
*GB
< gmaxwell>
We have docs that say 1.5GB, they're gonna be like the blocksize on bitcoin.org :)
< wumpus>
#7600 Mining: Select transactions using feerate-with-ancestors
< wumpus>
depends on what else is running on the machine
< gmaxwell>
I've been going through #7598/#7600.
< wumpus>
#topic compile-time memory usage
< wumpus>
what can *concretely* be done here?
< jonasschnelli>
would kick out boost help?
< luke-jr>
-O0
< wumpus>
is it something worrying?
< cfields_>
has anyone measured to see if there are particular objects that are especially guilty?
< CodeShark>
what's eating up all the RAM?
< wumpus>
yes, we have cfields_
< cfields_>
ie. main.cpp/net.cpp ?
< luke-jr>
CodeShark: ld/GCC doesn't free memory
< wumpus>
especialy some autogenerated c++ files
< wumpus>
I made some tables back in the issue about this
< gmaxwell>
main.cpp, matt has a patch that moves all the mempool stuff out of it taht apparently gets it back to 1.5GB.
< wumpus>
lots of people have posted about it, but there doesn't seem to be a clear solution
< jonasschnelli>
main.cpp -> 1248524bytes ... ^^
< wumpus>
reducing the number of included headers works, I think
< sipa>
present
< cfields_>
I have PRs which break up net.h/netbase.h, i'd be curious to see if those make a significant difference
< gmaxwell>
in any case, something to be aware of and nudge a bit at... some refactorings to move code around would help.
< wumpus>
also building with clang helps
< wumpus>
it uses a lot less memory at the same compile settings, usually
< gmaxwell>
and be independantly good for reasons unrelated to peak memory usage.
< cfields_>
i'd assume that mem usage correlates solidly with compile time
< CodeShark>
not so sure - lots of small files might mean the bottleneck is disk access
< sipa>
CodeShark: come on
< CodeShark>
in any case, it would be good to bring down the peak mem usage
< wumpus>
the bottleneck in compilation is hardly ever disk access, at least *reading* disk access
< sipa>
reading in 100 files?
< sipa>
sequentially
< BakSAj>
would be cool, if btc full nodes could continue to be runnable on Rasberry Pi ... with 1GB RAM
< jeremyrubin>
BakSAj: runnable is not compileable on?
< gmaxwell>
not sure there is much else to say here. I only brought it up for general awareness issues, since I think it's likely a death by 1000 cuts that can be improved in a multitude of ways.
< wumpus>
seek/read access for source files is only a problem for really huge projects, and then especially when the source is hosted on some horrible network file system (like clearcase), in any case bitcoin doesn't even come close
< cfields_>
heh, disk is negligible. It's easy to see where time is spent with -ftime-report.
< wumpus>
but like always: measure before you start talking about bottlenecks
< jonasschnelli>
I think adding cross compile depends options for ARM and AARCH64 would also reduce the "memory problem" (at least the amount of complains): https://github.com/bitcoin/bitcoin/issues/8162
< BakSAj>
jeremyrubin: preferably both compileable and operatable
< wumpus>
BakSAj: for small embedded systems you should use cross-compilation
< cfields_>
jonasschnelli: i'm halfway through the changes needed there.
< jeremyrubin>
i have had machines take a bit of time on autogen.sh fyi
< jonasschnelli>
cfields_: nice. Focus on Qt5.6 first. :)
< wumpus>
you can cross compile on ARM using depends, we just don't distribute ARM binaries
< cfields_>
jonasschnelli: actually, arm/aarch64 already work fine with depends. Just have to use NO_QT=1 manually.
< wumpus>
s/on/to
< gmaxwell>
I'm skeptical that the intersection of rpi users that complain about compile issues and people who will cross compile is the emptyset. But cross compiling is good.
< wumpus>
cfields_: yes, it works fine
< luke-jr>
for comparison, webkit-based stuff typically uses up to 12 GB RAM with debug symbols, and much much less without..
< gmaxwell>
er isn't the empty set, you get what I mean.
< cfields_>
luke-jr: oh, good point...
< wumpus>
it's *very easy* tocross compile for ARM
< jonasschnelli>
I think NO_QT=1 for ARM/AARCH64 could be a start (even for "official binaries").
< wumpus>
with the depends system
< wumpus>
jonasschnelli: yes
< cfields_>
the gitian-debug PR turns on debug symbols, so gitian mem requirement is bumped after that.
< gmaxwell>
wumpus: a lot of people using rpi2 like systems do not have another linux host.
< luke-jr>
wumpus: that builds static binaries, which is wasteful on RAM
< jonasschnelli>
Also ARM is used more and more for GUI systems.
< jeremyrubin>
can autogen.sh be made faster?
< wumpus>
jeremyrubin: no
< wumpus>
not by us, at least
< BakSAj>
ok, thanks for explaining.. personally i had no trouble compiling 0.12.1 on rpi 3, was afraid that minimum requirements will raise with future releases
< jonasschnelli>
luke-jr: if you want to run bitcoind on a RiP (or similar) static builds are fine. Mostly you don't have tons of other tools that could share libraries installed.
< BakSAj>
since suprisingly many nodes run on rpi
< jeremyrubin>
wumpus: maybe the one thing that is fixed by a faster disk
< luke-jr>
jonasschnelli: I'm thinking more of Bitcoin-Qt
< cfields_>
jonasschnelli: as qt's gui plugin situation improves, we may be able to move back to the shared-qt builds
< wumpus>
I'm sure minimum requirements will raise with future releases, that's just the way things are, we'll try to raise them not too much though
< MarcoFalke>
jonasschnelli: I think we already have notes on how to corss compile to arm?
< jonasschnelli>
Agree. Static linking qt is not ideal. But lets don't roll this up again.
< jonasschnelli>
MarcoFalke: notes, yes. But it should be included in our release builds (gitian)
< MarcoFalke>
jup, agree
< cfields_>
jonasschnelli: sorry, i meant that directly in the context of shipping arm+gui binaries
< luke-jr>
jonasschnelli: for now, people just compile natively to avoid static, so suggesting cross-compile isn't a real option
< wumpus>
jeremyrubin: I'm not sure. I have no idea what autogen.sh would be spending time on. But it seems more a GNU problem thatn a bitcoin core problem :)
< * gmaxwell>
looks forward to arm (+gui) binaries in the sometime future.
< wumpus>
jeremyrubin: I'm surprised it's autogen.sh taking a lot of time not configure, which has this huge list of scripts to execute for probing
< * luke-jr>
fully intends to use an ARM system with Core(Knots) as his hot wallet in some months.
< cfields_>
jeremyrubin: you can use a quicker shell for autogen. IIRC dash vs. bash shaves a few seconds off
< luke-jr>
autogen.sh isn't even part of building; it's a developer tool
< wumpus>
well arm non-GUI binaries would already be a great step forward, one step at a tme
< cfields_>
wumpus: ah, as a feature-freeze request: ok to plan on arm bins (without gui) for 0.13 ?
< luke-jr>
if you're running autogen.sh, that means you're running from git, and you shouldn't do that
< cfields_>
i can try to have that done today
< jonasschnelli>
cfields_: ack, +1
< wumpus>
I think arm gui would be prett much a per-distro afair
< wumpus>
cfields_: sure!
< luke-jr>
cfields_: +1
< jeremyrubin>
luke-jr: are we making build faster for developers or for users?
< gmaxwell>
+1
< cfields_>
ok
< luke-jr>
jeremyrubin: I think the concern is "ability to build" rather than "speed to build"
< gmaxwell>
jeremyrubin: users shouldn't need to run autogen-- if they get the source tarballs we have, it should already be autogenned.
< cfields_>
^^
< BakSAj>
cool, rpi fans will love you :-)
< wumpus>
but the people actually doing a lot of builds are developers, only they would care about a few more/less seconds in the build scripting
< BakSAj>
next step - run full node on cell phone :-)
< jeremyrubin>
wumpus: ++
< luke-jr>
BakSAj: I believe a number of people have done this.
< wumpus>
BakSAj: people are doing that actually, that's one of the motivations for the ARM binaries
< BakSAj>
lol ok
< luke-jr>
next step is therefore to support SPV mode when bandwidth is expensive ;)
< gmaxwell>
BakSAj: abcore, it works fine.
< jonasschnelli>
luke-jr: +1
< luke-jr>
but that's post-0.13 IMO
< wumpus>
absolutely
< wumpus>
in any case it's too late to start on anything new for 0.13, for that we have to consider which of the current pulls can go in
< luke-jr>
can we get in [8-bit] key generation type?
< jonasschnelli>
32bit!
< jonasschnelli>
You can provide a migration patch for Knots
< jonasschnelli>
Isn't that trivial?
< BakSAj>
will 0.13 contain just segwit code or actual softfork also? tnx
< jonasschnelli>
SW SF can be 0.13.1
< jonasschnelli>
SW are mostly not coupled with major releases
< jeremyrubin>
I think that 0.13.1 will be worse for upgrade times
< wumpus>
SW should be released in a minor release
< jeremyrubin>
does anyone have data on that
< gmaxwell>
BakSAj: major releases do not contain network consensus changes.
< sdaftuar>
do we think segwit is going in to 0.13?
< sdaftuar>
(without activation scheduled)
< jeremyrubin>
wumpus: isn't it a major change?
< petertodd>
sdaftuar: you mean, 0.13.0, or 0.13.x>0?
< luke-jr>
jonasschnelli: migration is not very practical if 32-bit uses the same version number in Core as 8-bit in Knots already is
< sdaftuar>
0.13.0
< btcdrak>
sdaftuar: yes, sipa wanted to merge it soon to master
< CodeShark>
what happened to doing it in 0.12.x?
< luke-jr>
jonasschnelli: maybe this needs more off-meeting discussion then
< btcdrak>
(without mainnet defs)
< jonasschnelli>
luke-jr: agree
< sdaftuar>
seems like there are still open issues, and no ACKs
< wumpus>
jeremyrubin: well from what I've heard minor releases are usually more popular, especialy .1, as some people don't trust .0 :)
< gmaxwell>
CodeShark: nothing, there are confused questions.
< sdaftuar>
so i don't see how it's going to be merged in the next week
< btcdrak>
sdaftuar: why?
< luke-jr>
jeremyrubin: segwit is a major change to Bitcoin - not Bitcoin Core.
< * jonasschnelli>
thinks sipa is allowed to merge without ACK
< gmaxwell>
jeremyrubin: we have a long thought out published spec on this, please don't divert the meeting to debating it. I can direct you to the information after the meeting.
< jeremyrubin>
wumpus: interesting... I do tend to not upgrade any of my software major versions for 6 months. diversion over
< sipa>
well merging segwit without fork enabled is not in contradiction with "not doing a consensus change in a major release"
< jonasschnelli>
agree
< CodeShark>
right
< wumpus>
sure
< luke-jr>
no objections to merging segwit code without activation
< jonasschnelli>
Also, getting ACK for SW is extremly hard. Nobody wants to take the risk.
< gmaxwell>
sure, there are code motion logistics that favor merging it.
< sdaftuar>
to be clear i'm not talking about any kind of release policy, just code-readiness / review
< wumpus>
so is SW ready for merge (into master/0.13)?
< sdaftuar>
it has no ACKs, and some open issues to be resolved
< wumpus>
ok
< jonasschnelli>
major open issue? Or more nitish stuff?
< sdaftuar>
minor
< wumpus>
if it is not critical it can also be fixed in a later pull
< sdaftuar>
but bugs, not style nits
< wumpus>
oh known bugs should be addressed in the pull itself
< sipa>
i think everything will be addressed in my next batch of patches
< btcdrak>
sipa: great!
< luke-jr>
sipa: does that include expanding 2nd push to 75 bytes max? or is that still an open thing?
< gmaxwell>
should people be acking the reviwew PR or the rebase/reorg?
< sipa>
luke-jr: this is the place to ask, and i would say no, there is no point
< sipa>
but perhaps others have another opinion
< btcdrak>
luke-jr: I didnt understand where 75 came from.
< sipa>
btcdrak: up to 75 is easy
< luke-jr>
btcdrak: largest size that wouldn't require additional testing
< gmaxwell>
has to do with the opcode types changing for different sizes of push.
< sipa>
so, opinions?
< btcdrak>
32->40->75 seems like a big jump
< gmaxwell>
btcdrak: from the code perspective they're all the same.
< luke-jr>
my opinion is there is no point limiting it (beyond the impl/test cost of >75), and such limits could very well prevent future softforks
< luke-jr>
more tolerant enables softforks, so should be preferred over useless limits
< gmaxwell>
Luke-jr's argument has merit in my opinion-- it can be reduced later, but I don't have a strongly held view. I'm not aware of a DOS attack risk created by not having the stricter limit earlier.
< gmaxwell>
(of course, IsStandardness should be strictly limited)
< luke-jr>
to expand the limit later requires a hardfork
< luke-jr>
yes, node policy should reject any unknown witnesses period
< CodeShark>
ok, I think luke-jr has a strong argument
< btcdrak>
that makes sense
< sipa>
there should be no need for more than 256-bit hash + some versioning metadata
< sipa>
and setting it to more gives it the impression that there is
< petertodd>
sipa: or, to be precise if there is that means Bitcoin is more broken than that
< sipa>
petertodd: exactly
< jeremyrubin>
luke-jr: in general I agree with keeping flexible, but do you have an example for sipa of why you'd want it?
< gmaxwell>
The biggest harm I see is that allowing a larger size here does limit the ability to make utxo entries limited in size in the future, potentially. But it could be done later. It also enabled policy bypass to abuse the utxo set for data storage, though it's not much of an issue there.
< luke-jr>
sipa: it doesn't need to give that impression. I don't think we need to predict the future too much here.
< gmaxwell>
luke-jr: for example, if you were to argue that we might someday need 512 bit hashes, I'd agree-- but then I'd point out that in that case there would need to be a hardfork to change all the other things.
< sipa>
i'd rather not rely on isstandardness when reasoning about longer term future
< petertodd>
in a MR implementation I did, it turned out to be very advantageous if the things in the MMR were fixed side forperformance
< luke-jr>
jeremyrubin: any case where we would need indicators in the UTXO set itself; but I don't have a concrete example at this time
< gmaxwell>
Also, not allowing it in SW doesn't preclude it in the future, you'd just need to use a different version type signaling in that case.
< luke-jr>
for example, we could have added the maturity stuff in the 2nd push if we didn't have nSequence
< gmaxwell>
Yes, I really wish UTXO entries were fixed size.
< sdaftuar>
sipa: isn't there a strong deterrent against abuse, because your funds are anyone-can-spend to older nodes?
< luke-jr>
gmaxwell: you'd need a new commitment entirely
< luke-jr>
gmaxwell: in addition to the current one
< sipa>
sdaftuar: there is no rule preventing 0-value outputs
< _anthony_>
just use the private key of a payment address to store the 256 bits
< sdaftuar>
ah, good point
< sipa>
(if you ignore relay polify)
< luke-jr>
abuse is already possible. this doesn't make it worse. if in the future we make it better, we can limit this at the same time
< gmaxwell>
if one assumes a fixed size utxo entry, luke's suggestion basically doubles the utxo set size.
< petertodd>
sipa: though, for that specific case I find it ahrd to think of a abuse use-case that'd care about that, given you could screw up the usse-case by spending those outputs
< luke-jr>
gmaxwell: we can't assume that today, and if we softfork an assumption tomorrow, we can limit this then also
< gmaxwell>
We've probably spent more time discussing it now than the decision is worth.
< wumpus>
ok, next topic?
< wumpus>
#topic compact block testing
< luke-jr>
so we use 75 for now, and discuss reducing it later?
< gmaxwell>
(and that time could be better spent reviewing/testing more corner cases... lets continue discussion elsewhere I guess)
< btcdrak>
so compact blocks...
< gmaxwell>
OK. So there are some number of nodes running compactblocks on the public network.. I have 12 peers at the moment, matt has another half dozen in the new relay network that I'm not connected to.
< gmaxwell>
Things seem to be working well there, instagibbs has posted some charts.
< wumpus>
I've been running a compact blocks node for a few days, no crashes to report :)
< gmaxwell>
I've been conducting some new tests with a network of nodes with a modified version of compact blocks that reduces the hash size to 16 bits in order to test corner cases around collisions.
< instagibbs>
blue stuff is in kB fwiw
< wumpus>
lots of succesfully reconstructed blocks
< luke-jr>
(ugh, Travis is apparently "detecting abuse" on the Bitcoin code itself, so every clone will be affected?)
< btcdrak>
Two large mining pools have also been running them, connected to their pool nodes for block source, one is behind the GFW
< instagibbs>
blue dot == 0 fetched txns
< gmaxwell>
I found a few bugs, which matt has fixed but not pushed to the PR yet. Bugs were things like if the cmptblk message was rubbish, it would wait for the peer to timeout before requesting the block normally.
< instagibbs>
I intended to review the PR then got ill. Still planning to review.
< gmaxwell>
I think this particular testing technique of modifying the code to make rare cases common is pretty effective and will result in good testing of most of those corner cases.
< wumpus>
luke-jr: (offtopic) that started happening with the parallel testing I think
< sipa>
gmaxwell: agree
< MarcoFalke>
luke-jr: Shoot them an email
< gmaxwell>
The compact block code is now rebased on top of the sharedptr work, so it's now a fair bit simpler.
< luke-jr>
MarcoFalke: I have. My concern is more than just whitelisting individual repos though. (Let's continue discussion after the meeting)
< instagibbs>
gmaxwell, matt's rebase is on that now?
< instagibbs>
err pr is rebased*
< gmaxwell>
instagibbs: yes.
< gmaxwell>
matt's PR is on master as of last night.
< cfields_>
has there been discussion of a servicebit for compact blocks? Now that we have the dns seed prefixes, that would allow for very quick discovery
< gmaxwell>
Based on the issues I found, probably the interaction with block fetching logic needs more review.
< btcdrak>
cfields_: if it deploys in 0.13 it wont be necessary
< gmaxwell>
cfields_: IMO I don't see a need to preferrentially peer. I expect support to become sufficiently ubiquitious fast enough.
< wumpus>
#action forget sipa's compact blocks branch and use thebluematt's PR
< gmaxwell>
it's not something that anyone has a reason to not support, except for just not having implemented it.
< btcdrak>
hrm, action point is to forget :)
< sipa>
cfields_: the argument brought up before was tgat service bits should be used for critical
< sipa>
for critically required services
< gmaxwell>
like your node won't work right if you don't have peers with the right services.
< wumpus>
btcdrak: yeah for people testing the code to use the other branch
< luke-jr>
makes sense
< sipa>
and the only time when yoi critically need a compact block peer is as a miner, who should be curating their connections anyway
< jeremyrubin>
in #8086 where is the salt generated btw?
< cfields_>
hmm, fair enough
< wumpus>
and miners can look at the protocol version to see if their peer supports compact blocks?
< luke-jr>
wumpus: I don't think we can assume a specific protocol version supports it
< luke-jr>
if we have a future version with better compact blocks, we may want to drop support for the current one
< jeremyrubin>
thanks
< Lightsword>
I think using service bits is a good idea, mainually curtailing connections is very time consuming and raisies the barrier to entry for mining
< gmaxwell>
wumpus: you can do the handshake.
< sipa>
wumpus: no, miners should connect to a known peer that supports it
< luke-jr>
Lightsword: neither are likely to be necessary
< wumpus>
gmaxwell: right
< gmaxwell>
Please, service bits are basically forever and we only have 32 of them, I expect the window between some and nearly all use of this to only be a few months to a year long.
< sipa>
wumpus: because just supporting compact blocks is not enough, they also need to have good uptime and reliability latency, bandwodth, ...
< sipa>
gmaxwell: we have 64
< gmaxwell>
Same difference. (really? hmph!)
< jeremyrubin>
I would suggest either writing the entropy to a file once or having it settable in a config file
< wumpus>
we should have a concept of temporary service bits, like for the versionbits
< sipa>
jeremyrubin: that's a good idea but orthogonal
< luke-jr>
as long as nobody relies on service bits, they can be temporary
< btcdrak>
we dont need preferential peering for compact blocks. It wont take long for wide network support.
< luke-jr>
ie, use them as hints
< cfields_>
don't we have a range designated for playground?
< luke-jr>
yes
< jeremyrubin>
sipa: (yes, sorry, just reviewing it now)
< Lightsword>
a service bit to indicate a secondary service bit field needs to be used?
< luke-jr>
one of which is currently getting full-RBF temporary usage
< wumpus>
Lightsword: that would completely make it unuseful for preferential peering
< gmaxwell>
jeremyrubin: uh. I'm not sure what you're talking about there... the nonces are per block and should not be predictable.
< wumpus>
Lightsword: (as neither addr messages nor the DNS seeds would be aware of the secondary mechanism)
< gmaxwell>
statically configuring it would be broken.
< wumpus>
why would you want to fix the entropy statically?
< instagibbs>
gmaxwell, perhaps setting cmpctblock as a tie-breaker for keeping connection?
< gmaxwell>
Okay, in any case, I think thats all I've got there.
< btcdrak>
ding ding, we have 7 mins remaining
< instagibbs>
well, I guess "he sent me blocks fast" is/will be one, same thing
< cfields_>
static entropy is much easier to test :p
< gmaxwell>
instagibbs: sounds like a fine additional ranker in the connection management stuff.
< wumpus>
instagibbs: +1
< sipa>
indeed
< jeremyrubin>
cfields_: yep
< gmaxwell>
instagibbs: though yea, the 'most recent blocks' probably mostly covers it.
< BakSAj>
which version are compact blocks planned for?
< sipa>
related to that: please review gmaxwell's patch for adding fast blkck and tx relayers for not evicted
< jeremyrubin>
gmaxwell: it doesn't harm security so long as it's kept secret from peers
< btcdrak>
BakSAj: 0.13.0
< instagibbs>
sipa, which number
< sipa>
instagibbs: sec
< jeremyrubin>
gmaxwell: nvm -- forgot you have to send it?
< gmaxwell>
jeremyrubin: the nonce used for compact blocks must be sent to peers or they can't recover the block.
< petertodd>
wumpus: we do have temporary service bits
< BakSAj>
btcdrak: thanks!
< sdaftuar>
gmaxwell: thoughts on #7598/#7600? you said above that you'd started review
< Lightsword>
isn’t it likely we’re going to overhaul the p2p protocol by the time we run out of service bits?
< sdaftuar>
i still think it should be a priority to get those PRs merged for 0.13.0...
< instagibbs>
I don't think connecting to cmpctblock peers will be hard unless we get sybil'd by AWS forks
< gmaxwell>
sdaftuar: I like them and will ACK soon, once I come up with a useful way to test.
< sipa>
sdaftuar: me too, i started revieweing but got caught up on other things
< gmaxwell>
sdaftuar: I agree.
< sipa>
Lightsword: maybe
< sipa>
Lightsword: it's often hard to predict how long protocols live
< gmaxwell>
I think important big PRs I'd really like to have in 0.13 are SW, Compact blocks, CFPF related, and BIP32.
< sdaftuar>
gmaxwell: ok, let me know if you want help with the sim environment i shared with you, i think that makes it easy
< gmaxwell>
There are a bunch of small things (including all of mine)
< sipa>
instagibbs: that one, thanks
< cfields_>
off-topic: quickly, before I forget. I'll be headed out of town on Friday and only reachable for emergencies for ~10 days. If anyone needs anything from me before I go, speak up now :)
< sipa>
cfields_: for how long?
< sipa>
a month?
< Lightsword>
maybe we should just have a service bit for flagging fast relay nodes/miners in general for preferential peering rather than making it flag compact blocks specifically
< wumpus>
only features have to be in before the feature freeze, anything that can be interpreted as bug fixes or anti-DoS measures doesn't have the deadline of next week
< wumpus>
also SW is special
< sipa>
Lightsword: we should also have an evil bit that abusive nodes should set
< gmaxwell>
Lightsword: "the I am a DOS attack master node, please connect to me" flag?
< Chris_St1>
brilliant
< btcdrak>
sipa: ^.^
< wumpus>
as we discussed above it's a consensus change so it can't be enabled in a major release first
< cfields_>
sipa: for ~10 days. I'll be gone for a month total, but working for the last few weeks.
< luke-jr>
FWIW, I will also be travelling June 20th-early July (but probably semi-available and working)
< sipa>
thanks all, this was an interesting meeting
< cfields_>
along the same lines, review plea for #8128
< wumpus>
instagibbs: I know, just wanted the situation around SW to be clear for people reading the logs
< instagibbs>
+1
< BakSAj>
btw, is there a rough estimate when 0.13.1 may come?
< BakSAj>
or version with segwit sf
< instagibbs>
Probably when a certain Softfork is ready....
< gmaxwell>
yea, my comments were assuming that it would be released in 0.12.x first, but parallel included in master.
< luke-jr>
BakSAj: "when it's ready"
< btcdrak>
BakSAj: SW will be released in 0.12.x
< gmaxwell>
BakSAj: 0.12.x
< cfields_>
luke-jr: btw, i believe I found your issue wrt out-of-tree build and chowning
< gmaxwell>
BakSAj: development operats long ahead of public release, we're often working up to a year ahead of whats released.
< cfields_>
luke-jr: we rely on "git diff" to update the index, but it can't if .git is read-only.
< luke-jr>
aha
< BakSAj>
uf little confused
< BakSAj>
now you release 0.13.0 with segwit code
< gmaxwell>
the interest in merging SW in master mostly comes from the fact that it isn't merged is holding people back from some changes because they don't want to make the SW rebases more complex.
< btcdrak>
gmaxwell: BlueMatt just pushed the fixes to #8068 compact blocks
< BakSAj>
but then 0.12.2 with code and activation comes
< instagibbs>
BakSAj, ok, so 0.13 isn't out. That means if SW is released, it goes into 0.12.2
< instagibbs>
then will still show up in 0.13
< gmaxwell>
For example, I wrote a patch to remove three loops over all transactions in accetblock... then shelved it, because its just a tiny speedup, and would conflict with the SW patches.
< BakSAj>
wont this bring confusion whether to run 0.12.2 or 0.13 ?
< instagibbs>
btcdrak, sigh, my block hit rate is going to poop :(
< cfields_>
luke-jr: could you verify that your process works if you add a "chown -R user:user" after the "chown -R root:root ." ?
< sipa>
BakSAj: what os are you running?
< luke-jr>
BakSAj: there is no one version everyone must run
< gmaxwell>
instagibbs: start up the relay node client for a bit to prefill your mempool.
< BakSAj>
sipa ?
< luke-jr>
cfields_: eh, that sounds like a noop?
< instagibbs>
gmaxwell, hmm ok pretty sure I have that installed
< btcdrak>
instagibbs: well you could spin up a new node instead...
< sipa>
BakSAj: what version of your os are you running.
< cfields_>
luke-jr: er, sorry...
< cfields_>
luke-jr: "chown -R user:user .git"
< BakSAj>
sipa: for full node?
< btcdrak>
instagibbs: rsync your chainstate etc.
< sipa>
BakSAj: whatever, i'm trying to make an analogu
< BakSAj>
lol, win10
< sipa>
BakSAj: you run 0.12.x to get more stable/tested code
< sipa>
BakSAj: you run 0.13.0 to get the latest and greates
< sipa>
and segwit is independent of that
< sipa>
it will be released in a point update to both
< gmaxwell>
SW is not a client feature, it's part of the network protocol.
< BakSAj>
i understand how sw livecycle usually works, but im confused about thing that 0.13.0 wont have SF active
< luke-jr>
BakSAj: if 0.13.0 is released before SW, it won't have SW; if it is released after SW, it won't remove SW
< gmaxwell>
BakSAj: it may or may not, but it won't be the first release with it. 0.12.x (likely 0.12.2) will.
< luke-jr>
SW introduction is independent from Core's release cycle
< sipa>
BakSAj: because in order to enable SW, we need many code changes
< sipa>
BakSAj: it's a pita to keep maintaining those in parallel when we know the code worms
< sipa>
*works
< BakSAj>
ah
< luke-jr>
BakSAj: if 0.12.1 and 0.13.0 are released before SW, then SW will be added in 0.13.1 and 0.12.2; if only 0.12.1 is released before SW, then it will be added to 0.12.2, and the later 0.13.0 will also have it still
< BakSAj>
i had bad assumption about 0.13.0
< BakSAj>
guys, thanks for explanation
< gmaxwell>
btcdrak: in the future, please wait for travis to finish before nagging people to update. :)
< instagibbs>
BakSAj, the section on the site about life cycle needs a refurb
< BakSAj>
if i can speak for little educated public :-) i would prefer complex btc roadmap
< BakSAj>
not just capacity increase roadmap
< btcdrak>
gmaxwell: noted
< dgenr8>
sipa: on .11.0 and .12.0 (I didn't try newer), after sendfrom "A", no subtraction from A is reflected in listaccounts or getbalance "A"
< dgenr8>
sipa: it looks like there is no way query ledger entries like this (or those created by move) :/
< dgenr8>
sipa: listunspent shows account A as an attribute of a utxo with address created by getnewaddress A. sendfrom A should probably wipe that out ... but that strays into trying to fix it
< sipa>
dgenr8: receive accounts are a property of incoming payments, not utxos
< sipa>
dgenr8: the fact that a send is not reflected in getbalance is a bug
< dgenr8>
I guess was speaking loosely ... i mean it has a txid and a vout, and it is unspent :)
< GitHub176>
[bitcoin] MarcoFalke opened pull request #8186: [0.12.2] backport: getblockchaininfo: make bip9_softforks an object, not an array. (0.12...Mf1606-rpcBip9Backport) https://github.com/bitcoin/bitcoin/pull/8186
< GitHub153>
[bitcoin] MarcoFalke opened pull request #8187: [WIP] [0.12.2] backport: [qa] Switch to py3 (0.12...Mf1606-qaPy3Backport) https://github.com/bitcoin/bitcoin/pull/8187