< jeremyrubin>
I think I found a bug in DynamicUsage
< jeremyrubin>
for unordered_map and unordered_set
< jeremyrubin>
The STL on my machine does an optimization where there's always one bucket (like a prevector)
< jeremyrubin>
This leads to double counting when you have a larger map
< jeremyrubin>
Trying to think of a x-platform way to deal with it...
< jeremyrubin>
this sounds like a thing for sipa to look at :)
< jeremyrubin>
This leads to a nasty-ish bug where when you're trying to track the internal memory of things, you can get an accounting deficit
< sipa>
jeremyrubin: DynamicUsage is inevitaby an approximation on some sysyems
< sipa>
improvememts welcome of course, but it's not possible in general (i think) to exactly know the size of data structures every STL implementation uses
< sipa>
i guess maybe with a custom allocator it could
< jeremyrubin>
One reasonable solution then is to use bucket_count -1
< sipa>
hmm, using a custom allocator to trakc this stuff should be easy
< jeremyrubin>
It will only ever be "off by one" so shouldn't matter too much, but would accurately report 0 allocation for a newly created set
< jeremyrubin>
sipa: I looked at that I don't think it is
< jeremyrubin>
but could be wrong
< jeremyrubin>
I guess something like bucket_count() - (bucket_count() == 1)
< jeremyrubin>
confirm that that approach fixes *my problem* :)
< luke-jr>
sipa: sizeof?
< jeremyrubin>
sizeof?
< luke-jr>
whatever size STL uses, you should be able to get usage using sizeof I would think
< jeremyrubin>
huh
< jeremyrubin>
I don't think so
< luke-jr>
why not?
< jeremyrubin>
So you want to know the size on the stack
< jeremyrubin>
you use sizeof
< jeremyrubin>
but what we want to know is the size of the allocation
< jeremyrubin>
And the reason it's difficult is because there's no way to tell if elements claimed by bucket_count() are on the stack or in the allocation
< jeremyrubin>
When you create a new unordered_map, bucket_count() is not nesc 0
< jeremyrubin>
e.g., implementations can optimize a pre-allocated bucket
< jeremyrubin>
that is a pointer for a single bucket that, once allocated, becomes the pointer for the allocation
< jeremyrubin>
or something like that
< luke-jr>
&(&object)[1] - &object ?
< jeremyrubin>
where object is what?
< jeremyrubin>
the map may be empty...
< sipa>
luke-jr: you don't know what the map allocates
< luke-jr>
hmm
< sipa>
it may allocate a struct { yourtype a; metadata b } object
< sipa>
or it may allocate those in arrays
< luke-jr>
C++ should add something :p
< sipa>
well, allocators
< sipa>
they let you observe the allocations STL containers make (and modify their behabior?
< sipa>
)
< luke-jr>
hmm
< jeremyrubin>
I think I heard there's an issue with allocators being assumed to be copyable or something funky like that, which can mess with collecting stats from one
< jeremyrubin>
But I don't recall what the issue is
< bitcoin-git>
[bitcoin] achow101 opened pull request #17261: Make ScriptPubKeyMan an actual interface and the wallet to have multiple (master...wallet-box-pr-2) https://github.com/bitcoin/bitcoin/pull/17261
< achow101>
ryanofsky: I remember now, the unique_ptr signingprovider commit was for descriptor wallets to work
< wumpus>
cfields: that's how I meant "giving up"; to find out that there's no elegent cross-platform solution for something in C++, and that boost works pretty ok for the case
< wumpus>
it's not preferable to boost to end up with maintaining reams of platform-specific compatibility code, or complex general algorithms such as date/time transformations
< wumpus>
maybe "Boost->C++11 migration" is just a misnomer,it's clear it's not possible yet, even with C++20 I think think it replaced all of our boost usage
< wumpus>
I *don't* think
< wumpus>
possilby it'd be better to close that as an active project and add "Don't use boost for new code, when remotely possible." to the developer notes and call it a day
< bitcoin-git>
[bitcoin] Sjors opened pull request #17264: [rpc] set default bip32derivs to true for psbt methods (master...2019/10/bip32_derivs) https://github.com/bitcoin/bitcoin/pull/17264
< wumpus>
#proposedmeetingtopic close Boost -> C++11 migration project for now
< sipa>
elichai2: it's a source of entropy on its own (by timing periodically during a slow computation), plus it strengthens the existing entrooy
< sipa>
as in: if there is not much entropy in there already, attacking will become a multiplication factor slower for an attacker to try anything
< sipa>
similar to pbkdf2 for passwords
< elichai2>
but these are passwords, usually words (easily bruteforcable with dictionaries). not sure attacking bad entropy of randomness is done via brute forcing (I honestly don't know. that's why I'm asking)
< elichai2>
I'm just wondering if adding complexity without obvious advantages is a good thing or not in the RNG world. but for me "entropy" is still a bit of a magic word so curious to hear your thoughts :)
< sipa>
entropy is actually the wrong term to use, as it's a concept from information theory
< sipa>
and we care about computational complexity for an attacker, not just whether the information is there in the first place
< sipa>
but say an attacker knows somehow that all the previous entropy sources we used only have 2^32 possible combinations, he can try all of them, and see if the output of the RNG for any of those seeds matches the observed behavior of the node (or say, generated keys)
< sipa>
and there strengthening could be the difference between possible and impossible
< sipa>
but it is defense in depth. if the entropy sources before work as expected, it is unnecessary
< elichai2>
but don't you decrease the entropy by introducing the chances of collisions of the PRF multiplied by the times you run the function on it?
< sipa>
no
< elichai2>
(i.e. SHA256 can have collisions even in 256bit inputs)
< sipa>
there is a very theoretical (and only relevant when you're actually talking about entropy and not computation necessary for attack) reduction in entropy when you have a 256-bit state with actually 256 bits of entropy, and you sha256 it repeatedly
< sipa>
and it only reduces things to slightly below 256 bits
< sipa>
however, that's not even what we're doing here
< sipa>
we extract 256 bits of entropy from the state, repeatedly hash it, and then mix the result back into the state... using a 512-bit hash function
< sipa>
how many bits the strengthening uses is pretty much immaterial; it could be just 64 bits too, probably... as we don't replace the entire state with the outcome
< sipa>
it just so happens that we have super optimized sha256 code, so we'd like to use it increase the advantage over an attacker
< elichai2>
That makes a bit more sense to me. I thought it's rehasing the data into itself. but looking again at the code it's obvious that's not what it's doing
< sipa>
no, it doesn't even hold the rng lock during the stengthening
< sipa>
other entropy may be added to it in the mean time too