< jeremyrubin> Hey
< jeremyrubin> I think I found a bug in DynamicUsage
< jeremyrubin> for unordered_map and unordered_set
< jeremyrubin> The STL on my machine does an optimization where there's always one bucket (like a prevector)
< jeremyrubin> This leads to double counting when you have a larger map
< jeremyrubin> Trying to think of a x-platform way to deal with it...
< jeremyrubin> this sounds like a thing for sipa to look at :)
< jeremyrubin> This leads to a nasty-ish bug where when you're trying to track the internal memory of things, you can get an accounting deficit
< sipa> jeremyrubin: DynamicUsage is inevitaby an approximation on some sysyems
< sipa> improvememts welcome of course, but it's not possible in general (i think) to exactly know the size of data structures every STL implementation uses
< sipa> i guess maybe with a custom allocator it could
< jeremyrubin> One reasonable solution then is to use bucket_count -1
< sipa> hmm, using a custom allocator to trakc this stuff should be easy
< jeremyrubin> It will only ever be "off by one" so shouldn't matter too much, but would accurately report 0 allocation for a newly created set
< jeremyrubin> sipa: I looked at that I don't think it is
< jeremyrubin> but could be wrong
< jeremyrubin> I guess something like bucket_count() - (bucket_count() == 1)
< jeremyrubin> confirm that that approach fixes *my problem* :)
< luke-jr> sipa: sizeof?
< jeremyrubin> sizeof?
< luke-jr> whatever size STL uses, you should be able to get usage using sizeof I would think
< jeremyrubin> huh
< jeremyrubin> I don't think so
< luke-jr> why not?
< jeremyrubin> So you want to know the size on the stack
< jeremyrubin> you use sizeof
< jeremyrubin> but what we want to know is the size of the allocation
< jeremyrubin> And the reason it's difficult is because there's no way to tell if elements claimed by bucket_count() are on the stack or in the allocation
< jeremyrubin> When you create a new unordered_map, bucket_count() is not nesc 0
< jeremyrubin> e.g., implementations can optimize a pre-allocated bucket
< jeremyrubin> that is a pointer for a single bucket that, once allocated, becomes the pointer for the allocation
< jeremyrubin> or something like that
< luke-jr> &(&object)[1] - &object ?
< jeremyrubin> where object is what?
< jeremyrubin> the map may be empty...
< sipa> luke-jr: you don't know what the map allocates
< luke-jr> hmm
< sipa> it may allocate a struct { yourtype a; metadata b } object
< sipa> or it may allocate those in arrays
< luke-jr> C++ should add something :p
< sipa> well, allocators
< sipa> they let you observe the allocations STL containers make (and modify their behabior?
< sipa> )
< luke-jr> hmm
< jeremyrubin> I think I heard there's an issue with allocators being assumed to be copyable or something funky like that, which can mess with collecting stats from one
< jeremyrubin> But I don't recall what the issue is
< bitcoin-git> [bitcoin] achow101 opened pull request #17261: Make ScriptPubKeyMan an actual interface and the wallet to have multiple (master...wallet-box-pr-2) https://github.com/bitcoin/bitcoin/pull/17261
< achow101> ryanofsky: I remember now, the unique_ptr signingprovider commit was for descriptor wallets to work
< wumpus> cfields: that's how I meant "giving up"; to find out that there's no elegent cross-platform solution for something in C++, and that boost works pretty ok for the case
< wumpus> it's not preferable to boost to end up with maintaining reams of platform-specific compatibility code, or complex general algorithms such as date/time transformations
< wumpus> maybe "Boost->C++11 migration" is just a misnomer,it's clear it's not possible yet, even with C++20 I think think it replaced all of our boost usage
< wumpus> I *don't* think
< wumpus> possilby it'd be better to close that as an active project and add "Don't use boost for new code, when remotely possible." to the developer notes and call it a day
< bitcoin-git> [bitcoin] laanwj pushed 12 commits to master: https://github.com/bitcoin/bitcoin/compare/25d7e2e78137...d91af4768c1c
< bitcoin-git> bitcoin/master 67328bb fanquake: build: remove protobuf from depends
< bitcoin-git> bitcoin/master 1cb9a4e fanquake: docs: remove protobuf from docs
< bitcoin-git> bitcoin/master 3548e4a fanquake: Remove BIP70 Support
< bitcoin-git> [bitcoin] laanwj merged pull request #17165: Remove BIP70 support (master...remove_bip70) https://github.com/bitcoin/bitcoin/pull/17165
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/d91af4768c1c...e9f73b839ad9
< bitcoin-git> bitcoin/master 6b6be41 Joao Barbosa: gui: Make polling in ClientModel asynchronous
< bitcoin-git> bitcoin/master e9f73b8 Wladimir J. van der Laan: Merge #17135: gui: Make polling in ClientModel asynchronous
< bitcoin-git> [bitcoin] laanwj merged pull request #17135: gui: Make polling in ClientModel asynchronous (master...2019-10-fix-gui-freeze) https://github.com/bitcoin/bitcoin/pull/17135
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to 0.19: https://github.com/bitcoin/bitcoin/compare/5b68d1654f07...0f6f7a574a4e
< bitcoin-git> bitcoin/0.19 d5c36ce Joao Barbosa: gui: Make polling in ClientModel asynchronous
< bitcoin-git> bitcoin/0.19 0f6f7a5 Wladimir J. van der Laan: Merge #17252: 0.19: gui: Make polling in ClientModel asynchronous
< bitcoin-git> [bitcoin] laanwj merged pull request #17252: 0.19: gui: Make polling in ClientModel asynchronous (0.19...2019-10-backport-17135) https://github.com/bitcoin/bitcoin/pull/17252
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/e9f73b839ad9...3f875744cb61
< bitcoin-git> bitcoin/master a592913 Jan Beich: http: add missing header bootlegged by boost < 1.72
< bitcoin-git> bitcoin/master 3f87574 Wladimir J. van der Laan: Merge #17249: rpc: Add missing deque include to fix build
< bitcoin-git> [bitcoin] laanwj merged pull request #17249: rpc: Add missing deque include to fix build (master...boost) https://github.com/bitcoin/bitcoin/pull/17249
< bitcoin-git> [bitcoin] Sjors opened pull request #17264: [rpc] set default bip32derivs to true for psbt methods (master...2019/10/bip32_derivs) https://github.com/bitcoin/bitcoin/pull/17264
< wumpus> #proposedmeetingtopic close Boost -> C++11 migration project for now
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/3f875744cb61...9bd109b78d93
< bitcoin-git> bitcoin/master a8f5026 Joao Barbosa: gui: Fix start timer from non QThread
< bitcoin-git> bitcoin/master 9bd109b Wladimir J. van der Laan: Merge #17120: gui: Fix start timer from non QThread
< bitcoin-git> [bitcoin] laanwj merged pull request #17120: gui: Fix start timer from non QThread (master...2019-10-fix-timer) https://github.com/bitcoin/bitcoin/pull/17120
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to 0.19: https://github.com/bitcoin/bitcoin/compare/0f6f7a574a4e...e39c9cff1a62
< bitcoin-git> bitcoin/0.19 8082b38 Jan Beich: http: add missing header bootlegged by boost < 1.72
< bitcoin-git> bitcoin/0.19 e39c9cf Joao Barbosa: gui: Fix start timer from non QThread
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/9bd109b78d93...be50469217bd
< bitcoin-git> bitcoin/master e156b9d fanquake: gui: disable font antialiasing for QR image address
< bitcoin-git> bitcoin/master be50469 Wladimir J. van der Laan: Merge #17257: gui: disable font antialiasing for QR image address
< bitcoin-git> [bitcoin] laanwj merged pull request #17257: gui: disable font antialiasing for QR image address (master...disable_qr_font_antialiasing) https://github.com/bitcoin/bitcoin/pull/17257
< bitcoin-git> [bitcoin] laanwj pushed 1 commit to 0.19: https://github.com/bitcoin/bitcoin/compare/e39c9cff1a62...eb5a899d5a97
< bitcoin-git> bitcoin/0.19 eb5a899 fanquake: gui: disable font antialiasing for QR image address
< wumpus> looks like things are ready, tagging 0.19.0rc2 in a bit
< JeremyCrookshank> :)
< bitcoin-git> [bitcoin] BlockMechanic closed pull request #17078: Android depends (master...android-depends) https://github.com/bitcoin/bitcoin/pull/17078
< bitcoin-git> [bitcoin] laanwj pushed tag v0.19.0rc2: https://github.com/bitcoin/bitcoin/compare/v0.19.0rc2
< fanquake> ?
< emilengler> ?
< bitcoin-git> [bitcoin] fanquake opened pull request #17265: [WIP] Remove OpenSSL (master...remove_openssl) https://github.com/bitcoin/bitcoin/pull/17265
< fanquake> pushed some rc2 sigs up https://github.com/bitcoin-core/gitian.sigs/pull/1118
< cfields> wumpus: forgot version bump? :)
< wumpus> cfields: oh no...
< wumpus> we really need a way to detect this
< wumpus> there used to be a time when the rc was auto-detected from the tag
< wumpus> I'm still not really used to this way
< wumpus> anyhow rc2 is dead, will do a rc3
< provoostenator> Long live rc3!
< wumpus> hehe
< fanquake> ?
< achow101> would it be possible to add a travis job for tags only that checks version number against tag?
< ghost43> you can do if: tag IS present
< wumpus> by that time it's too late
< wumpus> I think I'm going to add a script to maintainer tools to do the tagging, and also the version bump
< achow101> git pre-push hook?
< wumpus> meh
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to 0.19: https://github.com/bitcoin/bitcoin/compare/eb5a899d5a97...7358ae6d71cd
< bitcoin-git> bitcoin/0.19 f6aab42 Wladimir J. van der Laan: build: bump version to rc3
< bitcoin-git> bitcoin/0.19 7358ae6 Wladimir J. van der Laan: gui: rc3 translations update
< bitcoin-git> [bitcoin] laanwj pushed tag v0.19.0rc3: https://github.com/bitcoin/bitcoin/compare/v0.19.0rc3
< luke-jr> Subversion had a token that would be replaced with commit info in checkouts..
< luke-jr> I think git might have something similar
< luke-jr> can't remember the term to find it tho
< wumpus> I need to make a tagging script anyway (also to automatically set the Tree-SHA512:)
< bitcoin-git> [bitcoin] elichai opened pull request #17266: Rename DecodeDumpTime to ParseISO8601DateTime (master...2019-10-FormatISO8601DateTime) https://github.com/bitcoin/bitcoin/pull/17266
< bitcoin-git> [bitcoin] elichai closed pull request #17245: wallet: Remove Boost from DecodeDumpTime (master...2019-10-DecodeDumpTime) https://github.com/bitcoin/bitcoin/pull/17245
< bitcoin-git> [bitcoin] promag closed pull request #17133: 0.19: gui: Fix start timer from non QThread (0.19...2019-10-backport-17120) https://github.com/bitcoin/bitcoin/pull/17133
< bitcoin-git> [bitcoin] nijynot opened pull request #17267: bench: Fix negative values and zero for -evals flag (master...fix/eval) https://github.com/bitcoin/bitcoin/pull/17267
< elichai2> sipa: Hi, if you're free to expand on the logic/source of this idea I'm really curious to know about it :) https://github.com/bitcoin/bitcoin/pull/15224
< sipa> elichai2: it's a source of entropy on its own (by timing periodically during a slow computation), plus it strengthens the existing entrooy
< sipa> as in: if there is not much entropy in there already, attacking will become a multiplication factor slower for an attacker to try anything
< sipa> similar to pbkdf2 for passwords
< elichai2> but these are passwords, usually words (easily bruteforcable with dictionaries). not sure attacking bad entropy of randomness is done via brute forcing (I honestly don't know. that's why I'm asking)
< elichai2> I'm just wondering if adding complexity without obvious advantages is a good thing or not in the RNG world. but for me "entropy" is still a bit of a magic word so curious to hear your thoughts :)
< sipa> entropy is actually the wrong term to use, as it's a concept from information theory
< sipa> and we care about computational complexity for an attacker, not just whether the information is there in the first place
< sipa> but say an attacker knows somehow that all the previous entropy sources we used only have 2^32 possible combinations, he can try all of them, and see if the output of the RNG for any of those seeds matches the observed behavior of the node (or say, generated keys)
< sipa> and there strengthening could be the difference between possible and impossible
< sipa> but it is defense in depth. if the entropy sources before work as expected, it is unnecessary
< elichai2> but don't you decrease the entropy by introducing the chances of collisions of the PRF multiplied by the times you run the function on it?
< sipa> no
< elichai2> (i.e. SHA256 can have collisions even in 256bit inputs)
< sipa> there is a very theoretical (and only relevant when you're actually talking about entropy and not computation necessary for attack) reduction in entropy when you have a 256-bit state with actually 256 bits of entropy, and you sha256 it repeatedly
< sipa> and it only reduces things to slightly below 256 bits
< sipa> however, that's not even what we're doing here
< sipa> we extract 256 bits of entropy from the state, repeatedly hash it, and then mix the result back into the state... using a 512-bit hash function
< sipa> how many bits the strengthening uses is pretty much immaterial; it could be just 64 bits too, probably... as we don't replace the entire state with the outcome
< sipa> it just so happens that we have super optimized sha256 code, so we'd like to use it increase the advantage over an attacker
< elichai2> That makes a bit more sense to me. I thought it's rehasing the data into itself. but looking again at the code it's obvious that's not what it's doing
< sipa> no, it doesn't even hold the rng lock during the stengthening
< sipa> other entropy may be added to it in the mean time too
< sipa> s/entropy/randomness/
< bitcoin-git> [bitcoin] JeremyRubin opened pull request #17268: Epoch Mempool (master...mempool-experiments-2) https://github.com/bitcoin/bitcoin/pull/17268
< elichai2> sipa: Thanks :)