< wumpus>
it's also not completely unprecedented, for example Transmission torrent client has a turtle icon, which can be clicked to slow down, indeed meant when using the computer for other things
< gmaxwell>
well a 10ms sleep per 20 transactions would pretty much take care of it, I expect.
< wumpus>
although as an idea for bitcoin it's new, I think. THere has been a patch to add a button to disable networking completely but that kind of misses the point. That means a full catch-up has to be done when re-enabling
< wumpus>
sipa: hahaha
< gmaxwell>
thats old. :P
< wumpus>
I hadn't seen it yet
< gmaxwell>
damn repost supporters. :P
< gmaxwell>
okay, I'll permit it, since where I saw it was in a private msg from my SO,
< sipa>
i heartily endorse reposts. of things i hadn't seen.
< gmaxwell>
wumpus: I suspect that in the long run to really address the resource usage we'll need something like bittorrent's utp to transfer blocks during catchup and ibd.
< gmaxwell>
to enable darkness and hate in the world?
< gmaxwell>
some internal algorithim, presumably on hardware with fast FPUs (e.g. nothing you're that likely to use MUSL on) I presume that it's faster than a fixed point version of the same algorithim. Also hate.
< midnightmagic>
:-D
< gmaxwell>
I found this out from friends chasing a bug report in libtheora... which uses mmx.. and mmx and fpu share registers... and you have to toggle between them, and the library very carefully fixes up the flag between entry and exit. ... but it didn't know that it had to do so across _libc_ calls, particularly malloc.
< midnightmagic>
does smartheap still exist? I wonder if they do something similar.
< midnightmagic>
so.. threaded theora programs stomp on it?
< gmaxwell>
threaded is fine, OS saves the state betwen context switches.
< midnightmagic>
now I'm curious. Can you link a URL I can read?
< gmaxwell>
app calls libtheora, lib theora flips on mmx uses some MMX calls malloc does some more MMX flips off MMX OMG THE WORLD ENDS.
< luke-jr>
gmaxwell: I just had to write http://codepad.org/XT8ifBBp to avoid losing my browser session when I switched to 64-bit.. :|
< gmaxwell>
luke-jr: that looks like code for a a fast log.
< luke-jr>
does not inspire confidence in browser developers
< gmaxwell>
luke-jr: assuming a particular floating point representation (implementation defined behavior hurrah).
< luke-jr>
gmaxwell: a fast log? it's adjusting alignment
< luke-jr>
oh, the MUSL code
< gmaxwell>
the MUSL code.
< gmaxwell>
it's extracting the exponent and pulling off the bias.
< midnightmagic>
ohh.. you mean libtheora is failing to do mmx fixup properly, not musl libc?
< * luke-jr>
makes a long list of KDE 5 bugs
< gmaxwell>
midnightmagic: it's fixing up before entry and exit, that libc is using the fpu is a surprise, -- and in C++ you'd be pretty screwed since it's easy for objects to malloc behaind your back in virtually any line of code.
< gmaxwell>
theora's bug ultimately, but libc non libm using floating point is a surpise.
< wumpus>
a heap implementation using floating point is a surprise to me too. I think the advice should be extended from not using floating point for monetary values, to not using it for any kind of precise resource tracking...
< midnightmagic>
gmaxwell: I love interesting bugs like that.
< wumpus>
"You have 3.14159... bytes free!"
< gmaxwell>
(though I still have no idea how you could possibly write MMX using code in C++ if you can't call malloc without restoring the fpu flags first.)
< wumpus>
I don't think MMX is much-used these days, main reason because there are newer SIMD instruction sets but another reason is its weird re-purposing of registers, it not only makes it hard for developers to use it but let alone compilers... 3DNOW had a similar problem IIRC
< wumpus>
at its height it was used for a few well-contained matrix multiplication functions in games, too much risk of interfering with the rest of the code otherwise. Newer sets such as SSE have their own registers so doesn't have that problem.
< luke-jr>
I think I read in GCC's manual sometime that it doesn't use older stuff when AVX is available
< gmaxwell>
Yes, SSE2 mostly mooted MMX. ... though mmx vs fpu aren't the only fpu flags that could cause you problems with malloc using the fpu.. in particular the flags to control rounding modes might have unexpected effects. :P
< wumpus>
yes, I'm by no means defending the use of floating point in resource allocation... :P
< wumpus>
especially in cross-platform code. Ouch.
< wumpus>
it's seems like of those bit flipping tricks that's very clever but a bad idea to use in production code
< wumpus>
maybe it's a few % faster on some CPUs but the pain for maintainers... :)
< * wumpus>
wonders if gcc or clang has a flag to warn on floating point usage
< gmaxwell>
can't say I haven't done it myself; https://git.xiph.org/?p=opus.git;a=blob;f=celt/mathops.h;h=1f8a20cb4540255ffc4ea0a5f6716a31798c6a6f;hb=HEAD#l130 but it's behind a flag, and behind a bunch of warnings about non-portability, on some devices its more than a couple percent speedups. :)
< gmaxwell>
(and operating on floating point signals, not a random piece of floating point code in a sea of integer code)
< wumpus>
well that acts on float, as long as you can assume IEEE floating point formats I see nothing wrong with that
< wumpus>
and indeed it's operating on floating point signals, not integer numbers where off-by-one errors can cause a crash like memory allocation :)
< wumpus>
I think non-IEEE floating point formats (well there can be some contention around NaN and Inf and such, but I mean the global layout) are rarer than not-1-byte-chars these days
< luke-jr>
wumpus: what if your platform has IEEE floats that conflict with the integer endian? :D
< wumpus>
luke-jr: hmm that's a valid point
< luke-jr>
wumpus: NEON's non-IEEE stuff differs only in behaviour, not format, I guess?
< wumpus>
yes, "non-IEEE" in modern usage refers to "sloppy" math behavior to impmlement some operations faster, not the format
< wumpus>
or even if lacking some minor things that hardly anyone uses anyhow like signaling NaNs
< wumpus>
but which are great fun in network protocols, especially because no one expect themn and they cause a trap
< wumpus>
also the DoS-by-small-numbers of x86 denormalized floating point math has always surprised me. It's easy to think of floating point horror stories.
< luke-jr>
btw, I think my Chromium problems were possibly caused by Wikipedia :p
< luke-jr>
it erroneously documented that 32-bit aligned 64-bit integers to 8 bytes same as 64-bit does
< luke-jr>
when in reality GCC aligns them to 4 bytes
< * luke-jr>
wonders if this is actually GCC's bug, since SSSE3 also crashed quite a bit
< wumpus>
in structures? the normal gcc behavior is to algign to the type's size, on any arch
< wumpus>
though you can override it with a compiler flag
< luke-jr>
yes, in structures
< luke-jr>
x86-32 GCC aligns 64-bit integer types to only 4 bytes
< luke-jr>
even with the 64-bit compiler using -m32
< wumpus>
just tried it out and you're right
< gmaxwell>
freaky.
< wumpus>
on x86_32. On ARM32 it's 8 as expected.
< wumpus>
so please correct wikipedia :)
< luke-jr>
I did, this time
< wumpus>
in principle it makes sense, 32-bit x86 has (originally) no 64-bit memory loads, 64-bit integers are treated as two 32-bit halves
< wumpus>
but it would have tricked me too.
< * luke-jr>
wonders why the lower half of his IPv6 address changed
< wumpus>
yesterday I realized not even Amazon EC2 has IPv6 support yet. If there's anywhere you'd expect it to be used it'd be mass virtualized hosting
< luke-jr>
O.o
< * luke-jr>
wonders if that's what's holding Travis back
< wumpus>
that could be the case, on the other hand, Travis was even lacking localhost IPv6 support, that is not the fault of the hoster :)
< wumpus>
can we please move the serialization for float/double to serialize_unsafe.h?
< wumpus>
I was kind of scared until I relaized it's only used for local files (such as the fee estimator), not for network protocols, not for consensus structures
< luke-jr>
can we make it so it *can't* be used for non-local stuff?
< luke-jr>
probably easier to just make it portable I guess
< wumpus>
moving it to a header with a very scary warning will at least be a first step
< luke-jr>
I see no reason not to move it unless someone plans to do something bette
< luke-jr>
r
< wumpus>
making it portable is one thing. Accepting arbitrary float/double bit fields over the internet without potential crashes is not.
< wumpus>
well it's easy if you know a platform, but it's platform specific
< wumpus>
in any case putting it in a separate header with a big warning would help
< * luke-jr>
wonders if he will regret moving back to 64-bit
< luke-jr>
<1 day up and I'm already at 95% RAM use
< wumpus>
you could skip x86 64-bit and directly move ahead to the future, ARM64 :)
< * luke-jr>
was actually one of the earliest x86-64 adopters, but moved back to 32-bit in 2011 ;)
< paveljanik>
luke-jr, buffer, cache...
< luke-jr>
paveljanik: already excluding those
< paveljanik>
bitcoind with dbcache=16G? ;-)
< wumpus>
ok, just moving the serializers for float/double to a separate header file doesn't seem to be possible, it relies on a certain order of definitions
< luke-jr>
I haven't even rebuild bitcoin-qt yet
< wumpus>
compilng C++ with paralellism 10?
< luke-jr>
nothing being compiled :P
< sipa>
wumpus: make floating point serialization require nType == SER_DISK?
< wumpus>
I wonder if jtimon has any ideas with regard to isolating a consensus-safe part of serialization
< wumpus>
sipa: it'd work, I guess...
< wumpus>
another option would be to use a special converter to/from binary for those few places that use it with float/double
< wumpus>
something like FLATDATA, UNSAFE_SERIALIZE_DOUBLE
< wumpus>
anyhow I wasn't prepared to fight screenfuls of C++ compiler errors
< GitHub193>
bitcoin/master 30930e8 Wladimir J. van der Laan: test: Explicitly set encoding to utf8 when opening text files...
< GitHub193>
bitcoin/master 9298601 Wladimir J. van der Laan: Merge #8840: test: Explicitly set encoding to utf8 when opening text files...
< GitHub152>
[bitcoin] laanwj closed pull request #8840: test: Explicitly set encoding to utf8 when opening text files (master...2016_09_textfiles_locale) https://github.com/bitcoin/bitcoin/pull/8840
< GitHub160>
bitcoin/master da94697 jnewbery: bitcoin-util-test.py should fail if the output file is empty
< GitHub160>
bitcoin/master 90adfab Wladimir J. van der Laan: Merge #8836: bitcoin-util-test.py should fail if the output file is empty...
< GitHub19>
[bitcoin] laanwj closed pull request #8836: bitcoin-util-test.py should fail if the output file is empty (master...bitcoin-tx-no-empty-outputs) https://github.com/bitcoin/bitcoin/pull/8836
< luke-jr>
wumpus: Konversation; it's mostly due to swapping I think
< wumpus>
ok, never used that one, though my experience with KDE apps is that they use lots of memory. This is very old experience though.
< luke-jr>
gdb needs some way to report wtf is using so much memory
< sipa>
gdb?
< sipa>
how about top
< wumpus>
mine is currently IRSSI, running on an amazon EC2 nano instance, 512 MB ram of which 250MB used, of which 117 by IRC
< sipa>
irssi as well here
< wumpus>
that's virtual size not RSS, RSS is something like 10MB
< sipa>
167 MB virtual here
< * sipa>
remembers the joke "emacs == eight megabytes and constantly swapping"
< wumpus>
luke-jr: there are some heap debugger tools, I found the one in gperftools reasonanly useful. But that's per-process, not system-wide
< wumpus>
system-wide yes tools like gtop seem to do the job?
< wumpus>
-g
< luke-jr>
top just tells processes :p
< wumpus>
there's no gtop, htop is nice tho :)
< luke-jr>
RSS doesn't really work when swapping
< wumpus>
I also like "dstat -cdnpmgs --top-bio --top-cpu" a lot, it also shows the most i/o using process
< luke-jr>
9 GB used by KDE for trivial crap
< luke-jr>
fun
< wumpus>
welcome to modern linux distros :(
< sipa>
maybe try a different desktop environment?
< sipa>
over time i've moved to more lightweight ones
< wumpus>
so many background services and such for the GUI, it's almost like windows these days in that regard
< sipa>
all i need is a terminal and a browser anyway
< luke-jr>
wumpus: iotop works well
< wumpus>
yes same here
< wumpus>
even tend to use lynx as a browser sometimes, for viewing python docs and such
< luke-jr>
sipa: I tried a few and couldn't find any I liked :/
< wumpus>
to reduce mouse usage and such
< wumpus>
luke-jr: lxqt must be something for you
< * luke-jr>
kills all Chromium's tabs and gets it down to 13 GB (83%)
< luke-jr>
wumpus: yeah, tried that. the main thing I need is a WM, and lxqt doesn't do that :|
< wumpus>
GUI browsers use so much memory
< luke-jr>
yes
< luke-jr>
I wonder what it'd drop to if I quit it entirely
< wumpus>
for the ultimate minimalism you could run Weston :) A minimal wayland compositor, doesn't do much more than showing windows and being able to switch between them, with a launcher bar with one button: to start a terminal... hehe
< luke-jr>
10 GB (60%) with Chromium entirely dead
< luke-jr>
wumpus: but.. taskbar and systray and stuff :x
< wumpus>
yes, it has none of that, even more limited than 90's desktop environment,it's really meant as a base for embedded systems not for desktop usage, but a few hardcore people do
< sipa>
luke-jr: i haven't had a taskbar in almost 10 years
< sipa>
wumpus: heh, i should try that
< sipa>
xmonad isn't that much more
< luke-jr>
Windows NT worked with 16 GB RAM :<
< luke-jr>
MB*
< wumpus>
yes xmonad looks like something I should take a look at some day
< wumpus>
I guess you mean windows NT worked with 16 *MB* RAM?
< wumpus>
though like emacs it was probably swapping all the time :)
< sipa>
i think the first system on which i installed linux had 32 MB
< wumpus>
I don't remember anymore how much memory my 486 had, but it was very few, I remember reverse-engineering the BIOS to find the registers to map 384kB extra
< wumpus>
now we don't blink at that amount...
< luke-jr>
lol
< sipa>
i may misremember. it was a pentium 60MHz
< wumpus>
I had an extremely crappy PC, for a long time, which was a big driver for me to look at things like Linux in the first place
< * luke-jr>
figures out the trick to disable compositing
< wumpus>
no, I think that works out
< luke-jr>
not that it helped :<
< luke-jr>
load average: 48.20, 34.62, 24.88 <-- I didn't know it could get that high
< wumpus>
don't people use KDE for the fancy effects? :)
< luke-jr>
not I
< luke-jr>
turn all that crap off
< wumpus>
the first time I saw a wobbling window, or a workspace switch animation it seemed quite impressive, quickly after that it becomes a waste of time
< luke-jr>
heh, and a bandwidth eater when accessing remotely
< sipa>
wumpus: yeah, i used beryl and compiz and luminosity for a while
< wumpus>
luke-jr: yes, even if 'remotely' is localhost different VM
< sipa>
but it was mostly good for impressing windows users who made snarky comments about linux's UI :)
< luke-jr>
wumpus: well, I have to disagree there. I had no problems doing 3D gaming over VNC to a VM some years ago.
< wumpus>
sipa: hahah exactly!
< wumpus>
sipa: 'it should look like in the movies'
< luke-jr>
a VM within a VM even XD
< sipa>
also, i used enlightenment 17 somewhere in 2005
< luke-jr>
(KVM running PCSX2 with GPU and USB controller passthrough, viewed over VNC)
< sipa>
a decade before it was released, i think
< wumpus>
luke-jr: did that have a special protocol to route the 3D drawing commands?
< luke-jr>
wumpus: no, just VNC
< luke-jr>
Tight/JPEG encoding I guess
< wumpus>
luke-jr: eh GPU passthrough is essentially the same
< luke-jr>
wumpus: not really. that's just for the rendering :P
< luke-jr>
I didn't view on that GPU
< wumpus>
okay, right
< luke-jr>
someday when I have free time (haha) maybe I'll finish .hack XD
< midnightmagic>
dwm or fail :-P
< luke-jr>
apparently I'm going to have to find *something*
< luke-jr>
annoying, Qt5 has decided you need KDE to get any theming, so I'll end up losing that I guess
< sipa>
my window manager uses 7.7 MB of RSS
< sipa>
my terminal emulator 61 MB, Xord 112 MB, and firefox 987 MB
< sipa>
*Xorg
< wumpus>
luke-jr: you can also theme qt through gtk *ducks*
< luke-jr>
>_<
< luke-jr>
I think that needs GNOME running?
< wumpus>
I don't think so, Ubuntu uses it too and Unity is not GNOME
< luke-jr>
unfortunately, half the reason I want Qt is because GTK is terrible
< wumpus>
I think it's just a qt plugin that themes-like-gtk
< wumpus>
but it surpirses me to hear that Qt5 has no native theming ability anymore, that's kind of weird, I'm fairly sure qt5 is used a lot in embedded development and those companies all use their own theming.
< wumpus>
which doesn't say that much, a lot just gets hacked on
< luke-jr>
I think it has the ability, but it's not possible to configure it without a "platform module"
< sipa>
if you use mmap with MAP_FIXED and specify nullptr as address, are you afterwards able to dereference nullptr?
< luke-jr>
sipa: I think dosbox uses this?
< sipa>
assuming you make the compiler sufficiently dumb to not optimize things away that it believes are impossible
< luke-jr>
of course, C still considers it undefined behaviour
< luke-jr>
I think recent Linux has an option to block it too
< luke-jr>
since it's been used in exploits or something
< wumpus>
sipa: yes, if the OS allows that that's a valid way to mmap the null page
< wumpus>
I vaguely remember modern linux distros disallow it by default
< wumpus>
because it could be used to trick the kernel in some cases
< wumpus>
luke-jr: yes exactly
< wumpus>
C compilers will still regard nullptr as an invalid pointer, no matter what is actually mapped there, this can cause surprises on MMU-less platforms that map I/O registers in that range
< sipa>
sure, it's undefined behaviour to dereference a null pointer, i assume
< wumpus>
indeed
< wumpus>
so statements just disappear, sometimes at random,m depending on the mood of the compiler
< wumpus>
recompile and they may be there again. C is a jungle with its undefined behavior...
< wumpus>
(well recompile with a small change, maybe in a different function)
< wumpus>
your best bet is to do it in assembly, then use a volatile asm() block which the comiler isnot allowed to optimize away
< wumpus>
or even a function defined in an auxiliary .s file
< sipa>
c++ has even more undefined behaviour :)
< wumpus>
oh yes, I certainly didn't mean that C++ is any better in that regard
< sipa>
taking a reference to a nullpointer, and taking a pointer to it will in any sane architecture give you a null pointer again, but it is undefined
< GitHub8>
[bitcoin] czzarr opened pull request #8849: print P2WSH redeemScript in getrawtransaction if it s not a pubkey (master...print-p2wsh-redeemscript-in-getrawtransaction) https://github.com/bitcoin/bitcoin/pull/8849
< GitHub8>
[bitcoin] laanwj opened pull request #8850: Implement (begin|end)_ptr in C++11 and add deprecation comment (master...2016_09_beginptr_deprecation) https://github.com/bitcoin/bitcoin/pull/8850
< wumpus>
but it happens that a pull has tons of ACKs and I miss it, that's why I asked
< wumpus>
heh :P
< wumpus>
jtimon: thanks, done
< jtimon>
thanks, np
< GitHub100>
[bitcoin] MarcoFalke closed pull request #8633: Ugly hack to print out tests as they are run to mitigate travis timeouts (master...test-driver-hack) https://github.com/bitcoin/bitcoin/pull/8633
< jtimon>
#8337 would make #8493 easier to read/replace but the acks were before the last rebase...re-review (or new review) welcomed...
< GitHub192>
[bitcoin] MarcoFalke opened pull request #8851: [wallet] Move key derivation logic from GenerateNewKey to DeriveNewChildKey (pstratem) (master...Mf1610-walletDeriveNewChildPStratem) https://github.com/bitcoin/bitcoin/pull/8851
< wumpus>
jtimon: will take a look
< wumpus>
MarcoFalke: hah also looking at that pull
< MarcoFalke>
I caused the merge conflict so I felt responsible :)
< wumpus>
re: #8828 apparently ud2 is an opcode to explicitly generate a SIGILL "Generates an invalid opcode. This instruction is provided for software testing to explicitly generate an invalid opcode. The opcode for this instruction is reserved for this purpose."
< wumpus>
so that's deliberate, so much for accidental corruption of a function pointer, but why
< GitHub129>
bitcoin/master e198c52 Patrick Strateman: Move key derivation logic from GenerateNewKey to DeriveNewChildKey
< GitHub129>
bitcoin/master 940748b Wladimir J. van der Laan: Merge #8851: [wallet] Move key derivation logic from GenerateNewKey to DeriveNewChildKey (pstratem)...
< GitHub187>
[bitcoin] laanwj closed pull request #8375: [Wallet] Move key derivation logic from GenerateNewKey to DeriveNewChildKey (master...2016-07-19-cwallet-derivenewkey) https://github.com/bitcoin/bitcoin/pull/8375
< GitHub44>
[bitcoin] laanwj closed pull request #8851: [wallet] Move key derivation logic from GenerateNewKey to DeriveNewChildKey (pstratem) (master...Mf1610-walletDeriveNewChildPStratem) https://github.com/bitcoin/bitcoin/pull/8851
< wumpus>
paveljanik: congrats on finding the problem
< paveljanik>
wumpus, I was not able to parse the line mentally ;-)
< paveljanik>
I had to diff the left side and right side...
< wumpus>
paveljanik: I don't think anyone is, as it would involve a mental infinite loop
< wumpus>
but I hadn't noticed it
< wumpus>
anyhow the lesson is, if you do something as crazy as this and create a recursive reference, the compiler will generate an invalid instruction deliberately
< paveljanik>
I think this is a bug in the compiler.
< wumpus>
I don't know. How would you convert it to assembly if you were a compiler?
< paveljanik>
I'd error.
< wumpus>
it just makes no sense, garbage in, garbage out
< cfields_>
wumpus / MarcoFalke: before I forget: #8708 adds some sanity assertions that weren't there before. We were doing some things that didn't make sense. It immediately caused a few crashes for me, fixed in 89c5742. I suspect it will cause a few more assertion failures post-merge
< GitHub151>
bitcoin/master 203e2dd Lauda: Mention Gitian building script in doc.
< GitHub151>
bitcoin/master 7b784cc Wladimir J. van der Laan: Merge #8852: Mention Gitian building script in doc (Laudaa)...
< GitHub118>
[bitcoin] laanwj closed pull request #8852: Mention Gitian building script in doc (Laudaa) (master...2016_09_laudaa_master) https://github.com/bitcoin/bitcoin/pull/8852
< wumpus>
cfields_: makes sense
< wumpus>
cfields_: there's still oppertunity for that, 0.13.1 isn't out yet! :)
< cfields_>
wumpus: I figured you'd yell :). Doing now.
< wumpus>
paveljanik: I think overall that's a good point though, C/C++ compilers tend to raise errors only if something is absolutely disallowed due to syntax errors or such, if code makes just no sense and the compiler has no clue what to do with it, is apparently not enough reason
< wumpus>
cfields_: yes sorry, so many things to keep track of
< paveljanik>
compilers are afraid of errors. because their users would point to them... But sometime the bug is between the keyboard and the computer :-)
< cfields_>
wumpus: nono, I meant I figured you'd yell if I pr'd an addition to 0.13.1 at this point. I certainly wasn't expecting you to nag me about it.
< wumpus>
paveljanik: same for undefined behavior. Compilers just remove statements and instructions when something is undefined. If they were less sadistic they could just raise errors...
< paveljanik>
yes, definitely.
< wumpus>
cfields_: I don't know, I was surprised that 0.13.1 didn't come forward as topic at all in this week's meeting. I think between the segwit fix-ups you can certainly still sneak in network fixes
< GitHub161>
[bitcoin] czzarr opened pull request #8853: add p2sh and segwit options to bitcoin-tx outscript command (master...add-p2sh-segwit-options-to-bitcoin-tx) https://github.com/bitcoin/bitcoin/pull/8853
< GitHub180>
[bitcoin] czzarr closed pull request #8853: add p2sh and segwit options to bitcoin-tx outscript command (master...add-p2sh-segwit-options-to-bitcoin-tx) https://github.com/bitcoin/bitcoin/pull/8853
< wumpus>
huh
< wumpus>
this is already the second segwit-related pull that user opens accidentally against our repo and closes immediately: the other was #8816 4 days ago
< BlueMatt>
wumpus: same guy, meant to get private review before pr'ing to bitcoin core directly
< wumpus>
BlueMatt: ok :)
< morcos>
sipa: the optimistic write to pcoinsTip was a fantastic idea. thanks. i took a first pass at implementing it and it looks pretty good. i'm going to play with it some more.
< gmaxwell>
I think we all know whats needed for 0.13 mostly.
< BlueMatt>
what are the current thoughts on GetWitnessHash caching? (and before gmaxwell screams again, this isnt about 0.13.1)
< sdaftuar>
BlueMatt: i think we should do it, right? (you mean caching it in the CTransaction object?)
< BlueMatt>
sdaftuar: yes
< sdaftuar>
i guess it wastes memory for non-segwit transactions, but it seems to me like we'll want it in the long run
< gmaxwell>
Sipa has a PR that takes a good step towards making a flat const ctransaction object.
< gmaxwell>
so perhaps the fixed hashes should be done as part of completing that work-- making the objects flat will significantly reduce their memory usage.
< BlueMatt>
gmaxwell: heh, i was just looking at that...really hard to do efficiently since our serialization doesnt have length descriptors across the whole thing :(
< sdaftuar>
gmaxwell: i had a moment of brief panic the other day when i noticed the CTransaction::operator== function
< sdaftuar>
which just compares the stored hashes
< gmaxwell>
Well the flat object wouldn't be our seralization.
< gmaxwell>
e.g. you wouldn't want it to have varints in it. :)
< BlueMatt>
gmaxwell: suresure, but it still sucks since you cant prealloc the right size object from the start
< gmaxwell>
in any case, withash caching could just follow 8580 before going and flatting things. I expect flattening things will touch a lot of code.
< BlueMatt>
def need that for FIBRE ^
< BlueMatt>
ehh, that patch, that is
< gmaxwell>
BlueMatt: yea, it would end up doing something like deseralize into a mutable transaction then convert that to flat.
< BlueMatt>
gmaxwell: hmm, would that actually be faster?
< gmaxwell>
an extra copy, but it would save a half dozen or more calls to the heap allocator, and likely later accesses will involve less pointer chasing-- and the data will be smaller and more contigious. I would be fairly surprised if it wasn't.
< BlueMatt>
well the deserialization into a mutable transaction would do all the same heap allocations?
< gmaxwell>
hm okay indeed it would in the way I described it. Guess more cleverness would be required.
< wumpus>
q
< BlueMatt>
gmaxwell: yea, this is why i wanted to just use our serialization and copy to memory, but you cant really do that :(
< BlueMatt>
damn DRY serialization format :(
< gmaxwell>
well, not the end of the world. just deseralize into a maximum size buffer kept around for that purpose, then copy into the correct one.
< gmaxwell>
BlueMatt: if you want to see if you can fiddle with my compressed txn representation to make it easier to allocate for.. be my guest. Would be interesting if it could be done without losing too much packing efficiency.
< BlueMatt>
i mean if you want that just add a length descriptor as the first element over the entire tx (or anything you want to pack)
< BlueMatt>
it does ruin our nicely-DRY serialization, but it might be a valid tradeoff here
< gmaxwell>
other alternative is to parse twice to extra the length first, but I expect that to be slower than a tidy copy.
< BlueMatt>
quite possibly
< BlueMatt>
i mean if we do do a compressed txn format, we might consider this a feature, but I expect its hard to come up with something that can be trivially used for both without needing enough in-memory pointers and such that its no longer so useful
< gmaxwell>
oh I don't thin we can have one format for the compressed format and the flat in memory transaction. But we could try to make the compressed format easier to deseralize. E.g. by reading a small amount of it you would know how much to allocate for the flat form.
< gmaxwell>
s/thin/think/
< gmaxwell>
e.g. a format that coded all the sizes in the first few bytes. I would have naturally done that in my sketch, except I was trying to avoid having any bitpacking.
< gmaxwell>
but I think it could be reordered as is, so the it first gives all the type bytes and only after them does it seralize the payload.
< gmaxwell>
so a sizing parse would only have to handle the first O(txin+txout) bytes or so.
< BlueMatt>
yea, but if we have a nice in-memory blob format I can just make FIBRE use it :p