< Luke-Jr>
is it expected for valgrind to whine about libsecp256k1?
< gmaxwell>
Luke-Jr: more details required?
< Luke-Jr>
jumps based on uninit'd values
< gmaxwell>
do you mean the tests? yes--- if it's compiled with openssl then openssl taints the whole thing with uninitilized random numbers.
< gmaxwell>
Do you mean when compiled with clang? yes-- clang violates the C-spec usage for libc and will generate memcpy calls on overlapping areas.
< gmaxwell>
otherwise, no absolutely not.
< Luke-Jr>
k
< Luke-Jr>
(yes, it's with tests)
< gmaxwell>
you can compile without openssl to make those go away.
< gmaxwell>
I suppose I should make configure warn that this will happen.
< Luke-Jr>
it wasn't enough to bother me, unless it was indicative of a problem
< gmaxwell>
if you ever get code that has gone through my hands that emits valgrind warnings, please report.
< gmaxwell>
as an aside, if you compile your openssl with -DPURIFY it will stop executing undefined behavior and causing spurrious valgrind warnings in callers.
< Luke-Jr>
ugh, why is descendantfees not called descendantmodfees? :/
< Luke-Jr>
sdaftuar: ^
< sipa>
gmaxwell:
< GitHub67>
[bitcoin] luke-jr opened pull request #7529: Bugfix: Rename descendantfees to descendantmodfees (master...bugfix_descendantfees) https://github.com/bitcoin/bitcoin/pull/7529
< gmaxwell>
sipa: hm?
< sipa>
gmaxwell: ?
< sipa>
oh, i said something above
< sipa>
crap, i don't even remember typing that
< Luke-Jr>
lol
< warren>
Anyone had trouble with gitian-builder precise amd64 recently? It seems to install the VM just fine but it isn't bootable
< warren>
hmm, seems to be an issuer with newer versions of gitian-builder. older versions install a working VM but it gets stuck on ssh login due to some change elsewhere in the OS.
< warren>
"debug1: Skipping ssh-dss key ./var/id_dsa for not in PubkeyAcceptedKeyTypes"
< Luke-Jr>
morcos: why does CTxMemPool::UpdateForDescendants seem to be backward? ie, it's updating fields that should be on ancestors..?
< Luke-Jr>
warren: pseudo-fixed upstream, you need to redo the keying
< warren>
Luke-Jr: yeah found that, using git bisect to figure out what else changed between March 2015 and December 2015 that broke the VM install
< * Luke-Jr>
wonders why CTxMemPool code was rewritten obfuscated when there seems to be so many clearer possible ways to have done it :/
< phantomcircuit>
Luke-Jr, because it's keeping track of the opposite thing from cpfp
< phantomcircuit>
which ends up being weird
< Luke-Jr>
phantomcircuit: yeah, but I mean it should be UpdateForAncestors?
< Luke-Jr>
since it's updating the ancestor CTxMemPoolEntries
< Luke-Jr>
this code is all so obfuscated, I'm about to just go back to writing less-than-optimal CPFP code
< Luke-Jr>
or maybe I should just go update 0.11 :/
< wumpus>
cfields: made a mistake yesterday in buildling linux and mac, did not properly wipe the cache. Trying again...
< cfields>
wumpus: i just redid and got matches on both
< wumpus>
good
< cfields>
wumpus: if you don't match now, you're going to set my world on fire :p
< btcdrak>
music!
< cfields>
wumpus: ofc confirmation is most welcome. Still need another sig before signing though. After all that, I'm really not comfortable until we get 3rd party confirmation
< cfields>
tl;dr: nuke cache, fresh base image, rebuild windows
< wumpus>
cfields: yes, that must be it, and they upgraded the compiler for linux and mingw at the same time. I don't understand the macosx .tar.gz difference tho - does it contain a native executable somehow?
< wumpus>
cfields: oh, you rebuilt linux and it still matched?
< cfields>
wumpus: i must've not wiped properly
< wumpus>
my build yesterday was done on the wrong circumstances so I can't confirm linux and macosx stay the same
< cfields>
yikes, context needed there :)
< cfields>
wumpus: 2 fresh builds tonight of osx/linux, and they matched
< wumpus>
I apparently wiped the windows cache only :(
< wumpus>
cfields: ok
< cfields>
well, let's see how yours end up
< wumpus>
in my case no new base image was needed, the gcc upgrade is unavoidable :)
< warren>
cfields: the toolchain changing from under us is one reason why we eventually need to replace gitian
< warren>
(not to mention not fully knowing what's in that toolchain binary)
< cfields>
warren: yea, i think we're kinda purposely avoiding addressing the real issue here... that we're at the whim of the ubuntu toolchain packagers
< cfields>
but this is a good way to shed a light on that
< cfields>
(purposely for now, for the sake of getting the release out)
< warren>
well, for the purpose of getting the last 20 releases out =)
< wumpus>
warren: yes eventually the idea would be to build the toolchains too; I experimented a bit with bulilding specific toolchains myself using crosstool-ng. But as usual, other concerns came up and it went to the background a bit.
< wumpus>
I agree gitian *should* be replaced/improved to not rely on specific a linux distro eventually, but what we do is already much more than most projects do in this regard, and there isn't a replacement at the moment
< cfields>
wumpus: headed to bed. I'll check for matches in the morning. Please ping/mail me if anything comes up, I don't want to hold up the release
< wumpus>
cfields: ok nn
< warren>
wumpus: cfields and I have been talking for a few months about an eventual replacement, basic idea is to be able to build a deterministic toolchain on any Linux distro, then use that to build things like Bitcoin.
< warren>
wumpus: Then a one-time research effort could be done building toolchains from source from the oldest in history up to this deterministic toolchain and seeing if the result is identical.
< wumpus>
warren: yes, that's the basic idea. osx will, as usual, probably be hardest as it requries all kind of funny and weird tools built too
< wumpus>
building windows and linux cross-compilers is pretty straightforward
< warren>
yup
< wumpus>
(I've done it tons of times for embedded projects - it's mainly lots of waiting :-)
< wumpus>
(and lots and lots of harddisk space)
< btcdrak>
it would be amusing to know how many hours we've all spent in our lifetimes waiting for compiles to finish :p
< wumpus>
amusing but also unsetting :p
< wumpus>
there was an early commercial from a ISP in the netherlands about a guy that, having to wait so long for downloads, learned all kinds of skills like juggling and read all books inthe library etc, it feels the same sometimes :-)
< wumpus>
well I try to plan it usually that I don't have to wait for specific compiles by overlapping multiple
< cfields>
wumpus: in working on segwit, i had to mine some segnet coins after difficulty had been jacked up by some asic testing. That's like "I'm working, my code's compiling" x10 :)
< cfields>
wumpus: matches for osx/linux?
< wumpus>
oh yes the random and progress-less aspect of that makes it even worse
< wumpus>
linux is different :(
< wumpus>
c713f82859a27e9e0f9e7fb926f074bef855e255bfbb1207ce6987534233137a for bitcoin-0.12.0-linux64.tar.gz
< cfields>
sigh
< wumpus>
(different to my own previous build, at least, haven't checked with yours)
< wumpus>
trying mac now
< wumpus>
in any case the linux change is to be expected with a toolchain change
< cfields>
ok, nuking mine again and retrying
< wumpus>
although it makes no sense that you get the same as before
< wumpus>
worst case: the new gcc added some code randomization feature. OTOH we do tend to repeat our own results.
< cfields>
wumpus: there's a ton of room for pebcak atm. That's by far the most likely explanation for my results atm.
< wumpus>
yes just go to sleep we'll see tomorrow :)
< btcdrak>
sleep is a cureall
< cfields>
can't sleep when there's a mismatch and a result is pending :\
< warren>
cfields: if the underlying toolchain changes, perhaps the gitian-descriptors should be bumped so all cached results are invalid, to avoid expected mismatches?
< wumpus>
cfields: Good news: for osx, the dmg and -osx64.tar.gz stayed the same, just bitcoin-0.12.0-osx-unsigned.tar.gz changed. Same as you!
< wumpus>
cfields: my guess is that the -unsigned.tar.gz contains a locally built native executable?
< wumpus>
in any case, no problem for osx
< cfields>
wumpus: heh, so my macbook results were good, desktop were off. ok, that's something solid at least
< warren>
cfields: It's been over a decade since I've looked at it or tried it, but apt has pinning and particular versions of packages could be forced to stay that way and tested in a gitian-descriptor
< cfields>
wumpus: yea, the tools to re-attach and create the dmg are native
< wumpus>
we could use pinning for the compiler, on the other hand this is an ubuntu stable, if they do a compiler upgrade I'd expect them to have a very good reason
< wumpus>
cfields: okay, solved that then
< cfields>
warren: that's not a bad idea for 0.12, actually. the gitian descriptor can disable the cache
< cfields>
wumpus: maybe do ^^ and be done with it?
< warren>
horray I had an idea
< * warren>
sleep
< cfields>
doesn't solve any of the actual issues, but will help with coordinating
< warren>
cfields: I'm fighting a gitian problem for something else at the moment, so similarly annoyed
< wumpus>
cfields: yea, disabling the cache may be the safest, on the other hand having to rebuild -qt for every rc and minor release is a pain
< cfields>
wumpus: i was just speaking for 0.12 rc6/final, not anything permanent
< wumpus>
(and not once, five times - qt for win32, qt for win64, qt for linux32, qt for linux64, qt for macosx... did I mention qt takes a long time to build?)
< wumpus>
all the other dependencies are a joke in comparison and could as well be built every time
< wumpus>
I wonder if we build parts of qt that are unnecessary for us, but the cache convenience has always kept me from delving deeper
< cfields>
wumpus: i spent several days getting the qt build down to a bare minimum...
< wumpus>
okay
< cfields>
it might be better now with later versions, but in the ~5.2 days, that was about as slim as it could get
< cfields>
don't misunderstand, it's definitely worth taking another look. iirc at that point it wasn't split into separate modules
< cfields>
wumpus: but i'm discouraged by gentoo. Luke-Jr mentioned that they use similar hacks as us to build the native tools because it doesn't split up nicely
< Luke-Jr>
not sure how similar.. Gentoo just doesn't have them packaged separately
< cfields>
(ie there's no "qmake" ebuild, because it's not reasonable to build it without building the rest of base qt)
< wumpus>
cfields: I also don't expect them to have done much changes to qt in that regard
< Luke-Jr>
cross-compiling in Gentoo is somewhat of an afterthought
< warren>
wumpus, cfields: alternative idea, remember the old way of checking hashes if inputs in gitian descriptors prior to depends? Could do the opposite, include known hashes that it shouldn't be that cause the build to stop with an informative message
< warren>
Otoh I suppose versioning a set of expected tool chain is better
< cfields>
warren: yea, i'd prefer to avoid hard-coding because of the manual work involved. but in this case where the automatic deps failed us, static hashes start to look appealing
< cfields>
but i think we can work something out to avoid the static hashes
< warren>
The underlying tool chain changing should be something we detect
< wumpus>
cfields: match
< cfields>
wumpus: no idea what i borked last time, but went smoothly this time. building osx while i sleep. nnite now for real
< cfields>
warren: for sure
< wumpus>
warren: we can, all the .deb packages are logged to the .assert file
< wumpus>
warren: only caching messes with this :(
< wumpus>
the cache will have been built with a previous state of packages (or possible even multiple different ones over time), which isn't logged at this moment
< wumpus>
but the current state is always tracable and comparable
< wumpus>
if two people built with different gcc and have different output, normally we could see that. Except cache + unfortunate timing this time
< warren>
Idea: include the package list in the intermediate cached output tar
< wumpus>
all the gcc/g++ debs basically changed inbetween
< wumpus>
as long as *someone* builds without cache, these things will be detected
< wumpus>
I used to be that someone, but that was before having 6 rcs per release was all the rage :-)
< wumpus>
including a hash of the dpkg -l list in the cache hash (through a custom seed, as cfields suggested) may avoid this, although it will result in false positives
< warren>
Too many false positives, need to filter to a list of identified important tool chain dpkgs
< warren>
Ohhhh
< wumpus>
well it beats completely disabling the cache, but sure
< warren>
Use ccache as a canary, it's very good at detecting most toolcgain changes
< wumpus>
maybe a good idea - depends has support for ccache, but we explicitly disable that right now to rule out another potential issue due to caching
< warren>
I didn't mean use ccache for builds, use it only to detect changes
< wumpus>
yes I see
< warren>
It's a simple but imperfect detector
< GitHub39>
[bitcoin] knocte opened pull request #7530: autogen.sh: check for libtool before automake fails to find it (master...libtool-check) https://github.com/bitcoin/bitcoin/pull/7530
< wumpus>
I wonder if it would have detected this. I don't think gcc --version output changed :(
< wumpus>
otoh if it looks at, say, the timestamps of hashes of the actual tools then it may work
< wumpus>
(although gcc is sneaky in that regard, gcc etc is just a launcher it has a directory of actual tools stashed away somewhere else)
< wumpus>
anyhow an attempt at detecting this is 100% better than none
< MarcoFalke>
wumpus, Is your script to create the release note change log somewhere in the tree?
< PRab>
Any reason for the delay on rc5's detached sig?
< PRab>
I know this is overkill, but I think I'm going to blow away my entire VM and rebuild it.
< PRab>
I've been using the same one since I started doing gitian builds so I a couple of version behind for debian.
< MarcoFalke>
I think wiping the cache is just sufficient
< PRab>
Yeah, but I've been meaning to do this anyway.
< morcos>
Luke-Jr: I didn't write the UpdateForDescendants and associated code. But I don't really see why its necessary to be so critical about it.
< morcos>
It's difficult logic to get correct.
< morcos>
And the name to me seems fairly precise, it is well commented in the code. UpdateForDescendants updates the state of a particular transaction for any dependent txs already in the mempool.
< morcos>
I'm not sure what else you could call it. it isn't called on ancestors of a given tx, its called on all the txs that are readded to the mempool from a disconnected block
< morcos>
In any case, I'm sure if you were to rewrite the code better, no one would object, that would be more useful than complaining about its current form. I think you will find that task non-trivial.
< morcos>
sipa: regarding #7521 and attempting to reaccept/rerelay expired/evicted txs. although i still think not doing that is probably best, if we are going to do that, i think its important to add some timelimit.
< morcos>
there needs to be a dynamic way to recognize that some ancient tx from long ago with too little fee is just no longer viable in todays fee market/network
< morcos>
i've been thinking a bit about this current mass of giant spam txs floating around, and how we would ever get it to stop bouncing around the network endlessly
< morcos>
it seems reasonable to me that if you do want to try to reaccept txs that are no longer in your mempool so you can rebroadcast them, that you stop doing it after some period of time. a week, a month, something
< morcos>
measuring such time from tx creation, not the last time it succeeding in making it into your mempool.
< cfields>
MarcoFalke: strange, i'm unsure what's going on there
< PRab>
cfields: Building now.
< cfields>
PRab: great, thanks
< warren>
MarcoFalke: saw one of your github issues, you're the fedora user?
< MarcoFalke>
jup
< MarcoFalke>
I am doing the gitian building on the actual fedora host.
< warren>
MarcoFalke: ok, I made that vmbuilder and apt-cacher-ng package. upgrading gitian-builder to latest and manually populating the cache/ dir works around problems right now
< warren>
also need to rebuild the vm's
< MarcoFalke>
I am aware of the "rebuild the vm's"-fix because I broke it ;)
< warren>
FWIW I got gitian working on fedora last night by manually populating the cache with the intermediate source tarballs
< MarcoFalke>
warren, are you sure the vmbuilder hit yum yet? (Bug report shows it as "NEW")
< warren>
MarcoFalke: nobody approved the package review so it won't go into the repo. it was packaged over a year ago so maybe we should pull something newer from Ubuntu
< BlueMatt>
debug.log at 975G...quick someone ping my node a bunch so it hits 1TB :p
< morcos>
BlueMatt: shhh. they'll find out the secret reason we don't want to raise block size is we can't handle the larger debug logs
< PRab>
gbuild for windows doesn't appear to be finishing.
< PRab>
Its been hanging out at "Running build script ..." for over an hour.
< PRab>
(probably closer to 2)
< BlueMatt>
morcos: heh, well I tried to go to #btrfs to ask why their compression ioctl is screaming at me with no error message and the response I got was "well, go read fs/btrfs/ioctl.c"
< BlueMatt>
morcos: clearly the solution to the block size issue is to first fix btrfs
< Luke-Jr>
lol
< * Luke-Jr>
wonders if btrfs ever merged his bugfixes
< BlueMatt>
probably not
< Luke-Jr>
yeah, they don't seem to care about bugs :<
< morcos>
BlueMatt: btw, did you see 6977. its not an issue for fee estimation any more. but i assume we want the minreasonablerelay rate to change with command line option
< BlueMatt>
morcos: hmm, I havent seen that one...why does -minrelaytxfee continue to exist?
< BlueMatt>
morcos: like, did fee estimation think fee was 0 and thus min_fee?
< BlueMatt>
morcos: as for adding an option to set the min incremental relay fee, sure, why not...but should be a hidden option
< morcos>
BlueMatt: forget about fee esitmation for a second, it basically uses a new variable now anyway: fallbackfee
< BlueMatt>
ok, fair enough
< morcos>
The point is minrelaytxfee and incremental relay fee are the same thing now
< BlueMatt>
i mean then its the same thing - incremental fee for everything
< BlueMatt>
its also incremental from 0
< BlueMatt>
yea
< morcos>
right, thats fine, maybe they dont' need to be separate
< morcos>
but my point is if you try to set it (by setting minrelaytxfee)
< morcos>
it doesn't affect a couple of the calculations that happen inside mempool
< morcos>
b/c that variable is a local copy made before the command line option is set (i think)
< BlueMatt>
oh? well we should fix that
< BlueMatt>
we should also change minrelaytxfee to a different name
< BlueMatt>
and make it a hidden option
< BlueMatt>
because people who had previously set that as a spam-prevention thing should reconsider
< morcos>
Well, its not entirely clear
< BlueMatt>
should have done for 0.12, really
< BlueMatt>
but oh well
< morcos>
I mostly agree with that
< morcos>
but its nice to have the fall back of saying i just don't want any garbage less than X
< morcos>
for instance now you still get tons of useless traffic either when your mempoolminfee has temporarily decayed or because the mempool min never goes above 2 sat/B
< morcos>
i think your node in practice would actually be nicer behaved if you ran 0.11 with 5 sat/B minrelaytxfee right now. sadly.
< BlueMatt>
sat/B
< morcos>
but of course that just happens to be the exact traffic pattern on the network now. the dynamic solution is still the way to go, but its not a terrible idea to have a minimum which could be different from the increment
< BlueMatt>
oh, satoshi/Byte
< morcos>
yeah i don't know why we always say 1000 sat/kB instead of 1 sat/B
< Luke-Jr>
1 s/B is way too low a feerate :p
< morcos>
Luke-Jr: exactly :)
< Luke-Jr>
morcos: as for reason, it's because size used to be rounded to next kB
< BlueMatt>
i mean, sure, having a different incremental from 0 than incremental is a reasonable policy option
< BlueMatt>
but the real solution to the problem is to do some basic mempool-sync stuff on startup
< BlueMatt>
then your limit will jump right away
< morcos>
heh, the real solution is feefilter, its awesome in my testing so far
< morcos>
all these damn 15kB size txs of 15000 sat fee that keep being requested and then summarily rejected are really annoying!
< BlueMatt>
wow, scalia reported passed away
< BlueMatt>
thats gonna be big for scotus
< BlueMatt>
feefilter?
< BlueMatt>
I havent been paying attention of late, tbh
< morcos>
BlueMatt: its a tease, i haven't written it up yet. i'm procrastinating doing that now.
< BlueMatt>
whats the idea?
< morcos>
but short idea is you just tell your peers don't even bother inving me txs less than my mempool min fee
< BlueMatt>
ahh
< BlueMatt>
meh
< * Luke-Jr>
wishes there was somewhere better to move
< morcos>
whats the meh
< BlueMatt>
that doesnt help if your mempool never fills enough to get medium-fee txn
< BlueMatt>
so that you can set the mempool fee
< BlueMatt>
but, yes, we should also have that
< BlueMatt>
maybe we should revisit segwit having explicit fees
< morcos>
whose mempool never fills enough
< BlueMatt>
mempools do take a while to fill reasonably
< BlueMatt>
because people dont rebroadcast "real" txn
< BlueMatt>
just the spam keeps flowing around
< morcos>
yes, whats typical now is your mempool min fee is 2 sat/B and then slowly decays til you can start accepting the spam again
< Luke-Jr>
[22:11:37] <BlueMatt> because people dont rebroadcast "real" txn <-- ⁇
< morcos>
but during that decay, you are inved spam, and rereuest it b/c your reject filter is reset
< morcos>
feefilter fixes that
< BlueMatt>
Luke-Jr: i mean because it gets mined, so they dont have to :p
< BlueMatt>
morcos: yes, both are required, is my point
< morcos>
sorry, what's the both?
< morcos>
you mean dynamic mempool fee?
< morcos>
of course
< BlueMatt>
both mempool sync and feefilter
< morcos>
oh sync
< BlueMatt>
ofc sync is hard to do in a privacy-preserving manner, etc, etc
< BlueMatt>
we need to redo txn flow completely, really
< BlueMatt>
to preserve privacy
< morcos>
yeah, i'm not sure what i think about sync.
< morcos>
yeah well did you read the privacy concerns around 7521
< morcos>
its pretty bad
< BlueMatt>
do we really still do that?
< BlueMatt>
wow
< BlueMatt>
greg's proposal of regular-sync + send txn as you accept them to mempool to a fixed peer or two seems the most reasonable I've heard
< morcos>
not sure i heard that, why does the peer need to be fixed
< BlueMatt>
same reason rotating your tor guard node regularly is shit
< BlueMatt>
if you dont rotate your guard node/tx announce node, you have a fixed chance of being deanonymized...if you do, you are guaranteed to be deanonymized eventually
< BlueMatt>
with the same chance at each rotation
< morcos>
too many problems.. anyway, ok family time
< BlueMatt>
yea, talk to the tor guys....everything you try here fails :(
< morcos>
BlueMatt: You got me thinking about deanonymization with this feefilter thing. Is one of the problems I need to always be solving for making it difficult to correlate a tor node with a public node
< morcos>
In other words, if my node connects on both networks, i need to make sure someone that sees it on one can't identify it as the same node on the other?
< morcos>
The simple feefilter implementation would make that trivial as it would basically announce every time your mempool min fee kicks in or decays to 0.
< morcos>
Of course you could add some variance to obfuscate to some degree.
< BlueMatt>
I'm not really sure if thats in our threat model or not...I think it should be within reason, but I'm sure we break that all over the place right now
< morcos>
But actually even without the fee filter, a limited mempool makes it pretty easy to correlate (with a bit more work)
< BlueMatt>
yea
< BlueMatt>
kinda
< PRab>
Build finally finished. I guess it wasn't stuck, it just took a long time.