< shmulim>
there may be other unused dependencies, this commit does not represent a comprehensive check..
< luke-jr>
shmulim: just because it runs without them, doesn't mean they're unused (although in this case they are)
< sipa>
shmulim: well if we're going to go through an effort to clean up unused dependencies, it would be nice to know it's xomprehensive
< sipa>
*comprehensive
< luke-jr>
would be nice if GCC/LLVM had a way to hide the stuff inherited from includes, such that it can at least detect missing dependencies
< shmulim>
it is not comprehensive.. i'll leave a note on the PR.. thank you both for the pointers.. i'll look into iwyu.
< eklitzke>
I am kind of confused about how CheckInputs works. since the script execution cache can return false positives, there must be a path where the scripts are checked without checking the cache. can someone explain how that works?
< sipa>
eklitzke: how can it return false positives?
< sipa>
all the context, data, and flags for validation are hashed cryptographically into the cache's key
< sipa>
so at worst there is a discrepancy, and the entry just won't be found
< eklitzke>
is it not a probabilistic data structure?
< sipa>
it's probabilistic in the other direction
< eklitzke>
i see
< sipa>
and if you accept the chance of hash collisions on sha256, it also has a chance for false positives
< sipa>
but if those are a concern, we have bigger proble.s
< eklitzke>
i'm assuming that the problem of finding a sha256 collision is more difficult than finding a collision in the cache
< sipa>
hmm?
< sipa>
the cache stores the full hash
< eklitzke>
ah
< sipa>
its location can have collisions, in whicb case an entry will be evicted
< sipa>
but that still only results in a false negative
< eklitzke>
i had difficulty finding the original PR, it looks like the one that was merged on github is not the first PR put up (which probably has more of these details)
< eklitzke>
actually i might be wrong, maybe this description is using "original PR" to refer to the unsquashed version
< eklitzke>
it's not really like a bloom filter because the original element is retained, and contains() does an == check if it finds an item in the cache, which is your point about it being probabilistic in the other direction
< eklitzke>
so it just relies on sha256 of the item in the cache being unique
< sipa>
it does a few interesting things
< sipa>
every element can be in 8 different valid locations
< sipa>
and it gets moved around to make space for new ones, unless a cycle forms
< sipa>
and when an item is marked as expired, it it retained in the cache (and findable) but just available for overwriting
< bitcoin-git>
[bitcoin] luke-jr opened pull request #12674: RPC: Support addnode onetry without making the connection priviliged (master...rpc_onetry_nonpriv) https://github.com/bitcoin/bitcoin/pull/12674
< eklitzke>
i just wrote some code to dump the utxo database, the compression stuff is wild
< eklitzke>
i had no idea so much weird stuff went into making the values small
< sipa>
eklitzke: it started out as an experiment in seeing how small the utxo set could get
< sipa>
and ended up being the actual database format
< sipa>
look at how the txouts were encoded before per-txout, it close to simiarly crazy
< eklitzke>
i was originally just trying to get the height of everything in the utxo database, but once i saw the serialization code for amounts i had to add it
< eklitzke>
what is the motivatoin for obfuscating values with the obfuscate key in the leveldb databases?
< sipa>
av software :(
< kallewoof>
sipa: rumor has it overkill is your middle name
< eklitzke>
strange
< achow101>
eklitzke: some transactions in the blockchain contain virus signatures which AVs will catch and then remove from the chainstate database thus corrupting it. So obfuscating the data in the db prevents that from happening
< eklitzke>
huh, that is really fascinating
< achow101>
AVs are routinely a source of problems with running core in windows. More recently it's been AVs blocking connections to and from port 8333
< Randolf>
Anti-spyware software is getting more aggressive against anything that looks like it could be doing cryptocurrency mining. And unfortunately it's often classified as Bitcoin Mining Riskware.
< kallewoof>
I did some time profiling on current master (ran for about 30 mins) and found an oddity. May not be important, but 14.4% of the CPU time (9.48s) was spent in CCoinsViewDB::Upgrade(), in the pcursor->Seek call at the top (which is used to determine if an upgrade should happen or not).
< Randolf>
Sadly, many users haven't even a clue that it's possible to add exceptions into their anti-virus and anti-spyware software's configuration.
< kallewoof>
Also noticed 45.4% of the time is spent in leveldb::. More than main thread (main thread = 32.4%)..
< eklitzke>
did you filter out the bgthread
< kallewoof>
eklitzke: No filtering, no.
< eklitzke>
leveldb creates a background worker thread (you'll see BGThread in the stack trace) which tends to use a lot of cpu/disk time
< eklitzke>
it's kind of confusing because it's a thread created by leveldb itself, not managed by bitcoin
< kallewoof>
Hum. I tried adding snappy support and leveldb dropped a ton. Still need to verify results, but looks pretty interesting.
< eklitzke>
what platform?
< eklitzke>
there is something broken with how we're using the block cache right now (which is actually affected by whether you have snappy compression on or not), waiting to get my max_open_files change in before sending out that pr
< eklitzke>
in general though i would not expect snappy compression to help since the keys and values are already compressed
< kallewoof>
eklitzke: Yeah, I'm seeing roughly the same performance now. Good point on already compressed.
< kallewoof>
Actually I think performance dropped a bit with snappy, but very imprecise measurement (total CPU time over total time for separate runs).
< George_e30>
Hello everyone!
< bitcoin-git>
[bitcoin] dexX7 opened pull request #12676: Show "replaceable" flag, when decoding raw transactions (master...rpc-raw-replaceable-flag) https://github.com/bitcoin/bitcoin/pull/12676
< George_e30>
I install b2x-node, connect to it with stratum, and try to mine. But I have error in log:
< contrapumpkin>
launching 0.16.0 for the first time (upgrading from 0.15.1) and it thinks for a while then shows a scary error that a fatal error occurs and dies. The log says "ReadBlockFromDisk: Deserialize or I/O error -CAutoFile::read:fread failed: unspecified iostream_category error at CBlockDiskPos(nFile=1207, npos=133025664)". I assume this just means some of my block data is somehow corrupt? Can I pass in a magic flag to reuse as
< contrapumpkin>
much of it as possible while redownloading only the broken stuff?
< contrapumpkin>
right now it just refuses to do anything at all and quits after displaying the fatal error
< esotericnonsense>
you can use -reindex and that should use as much as possible of the non-corrupt data I believe. I'm not sure there's a more efficient way to do it
< esotericnonsense>
whether there's8
< eklitzke>
by my math 0.4% of utxos are 1 day old (or newer), does that sound approximately right? just trying to sanity check my results
< sipa>
that seems plausible
< sipa>
that's around 200k UTXOs
< eklitzke>
yeah that seams reasonable given 144 blocks per day
< sipa>
or 1400 utxos per block
< eklitzke>
i'm trying to figure out for a new utxo, the probability distribution of it being spent within N blocks
< jimpo>
eklitzke: Over what range of UTXOS (like UTXOs created since when)?
< eklitzke>
that's the probability distribution for any utxo being at a given chain height
< eklitzke>
it's cool i can dump the utxo set in a minute or two
< jimpo>
neat. y-axis is probability that UTXO is spent within a day?
< eklitzke>
it's the proportion of utxos at that chain height
< jimpo>
er, right
< bitcoin-git>
[bitcoin] luke-jr opened pull request #12677: RPC: Add ancestor{count,size,fees} to listunspent output (master...listunspent_ancestorinfo) https://github.com/bitcoin/bitcoin/pull/12677
< cfields>
<sipa> sdaftuar: seems my benchmark was off, and the jacobi symbol implementation in GMP takes 2us rather than 0.2us
< cfields>
:(
< cfields>
sipa: i guess you're ready to bail on EC for set hashing, then?
< cfields>
oh, I guess so if ECMH == elliptic curve multi-set hash
< sipa>
cfields: well 9.3us isn't horrible
< sipa>
it means maybe a 20-30% CPU increase overall for full validation, though
< cfields>
sipa: is it the case that gmp's jacobi impl isn't as far away from others as you had expected? Or would it still be substantially worse if we rolled our own?
< sipa>
cfields: so, there are two ways of doing hashing onto curves
< sipa>
A: do { x = generate 32 bytes; compute y=sqrt(x^3+7); if it exists return (x,y); }
< sipa>
B: do { x = generate 32 bytes; compute jacobi(x^3,7); if it is 1, return (x, sqrt(x^3+7)); }
< sipa>
given that around 50% of x values are acceptable, you on average need 2 iterations
< sipa>
so method A does 2 sqrts, method B does 2 jacobis + 1 sqrt
< sipa>
in other words, B is better than A when jacobi is more than 2x as fast as a sqrt
< cfields>
ok
< sipa>
with sqrt=5us and jacobi=0.2us, B was clearly worth it
< sipa>
but it seems that was wrong, and gmp's jacobi is more like 2us
< cfields>
ok, i'm with you now. thanks.
< sipa>
however, with gmp not being as crazy fast as i believed, it may be feasible to write a jacobi implementation that is faster than GMP
< bitcoin-git>
[bitcoin] vasild opened pull request #12678: Scripts and tools: Fix a few compilation issues with Clang 7 and -Werror (master...master-compilation-fixes-with-clang7-werror) https://github.com/bitcoin/bitcoin/pull/12678
< sipa>
we had previous given up on that, as 0.2us seemed impossible to beat
< cfields>
heh yes, i see now why it was written off so easily
< sipa>
but it is diminishing returns... we're not going to beat 6us or so overall
< sipa>
muhash would overall run in maybe 3-4us per update
< sipa>
so it's only a small constant factor difference
< cfields>
ah, i figured it was more substantial than that. ok, so a ~2x slowdown is reasonable to consider.
< intcat>
(not sure if correct channel to ask) i am trying to set up bitcoin core gitian build for 0.16.0, but dependency resolution fails on expat; it attempts to download from https://bitcoincore.org/depends-sources but gets 404
< cfields>
intcat: it tries to download from the upstream source first, and only hits bitcoincore.org if that fails. Most likely you don't have net access inside your container/vm.
< intcat>
cfields: actually, yes i do
< intcat>
a lot of other depends are downloaded correctly before this one
< cfields>
intcat: sec, let's check if upstream is down atm
< intcat>
i noticed master has a different upstream url than 0.16.0 tag
< intcat>
sourceforge vs github. and that kind of thing can happen, but it seems that the fallback is failing too
< cfields>
intcat: not sure if it's down, but I'm fixing up the links at bitcoincore.org anyway, since it needed to be done
< intcat>
:)
< intcat>
cfields: seems the container im trying to build in borks on the sourceforge redirect
< cfields>
intcat: not surprising, sf redirects several times
< intcat>
running the same command on another machine works, though
< intcat>
but still, would be nice to have working fallback