< dcousens>
the *.dat files, they are assumed to be in order yeah? I know the blocks in them may be out of order, but the files themselves are in order yeah?
< sipa>
what does order for files mean, if the blocks are not in order?
< dcousens>
sequential*
< dcousens>
in that, data from blk001.dat "flows" into blk002.dat
< sipa>
yes
< sipa>
:)
< sipa>
though that has been broken in the past
< dcousens>
sipa: oh?
< dcousens>
(having troubles with a parser, it makes it 370k blocks in before blowing up with the magic header being wrong)
< sipa>
it would at startup rescan through all block files the first time a block was written, and if there was space in one, it would go there
< dcousens>
(running a re-index now)
< dcousens>
sipa: is there a separate index for the blocks in the .dat files?
< sipa>
that's called the block index :p
< sipa>
$DATADIR/blocks/indx
< sipa>
$DATADIR/blocks/index
< dcousens>
haha
< dcousens>
Sure, is that referencing file pointers?
< dcousens>
like, file+offset pointers
< sipa>
indeed
< sipa>
file number, byte offset
< dcousens>
and even then, the blocks would still be back to back right, there shouldn't be empty space out of order data etc?
< dcousens>
like, I wouldn't expect a block to flow from 001.dat to 003.dat,
< dcousens>
(that is, skipping 002.dat)
< sipa>
that's certainly possible
< sipa>
there are really no guarantees
< sipa>
we try to put blocks in order
< sipa>
to the extent possible
< sipa>
but only by limiting how much out-of-order blocks we fetch
< dcousens>
sipa: wouldn't that mean the block index would have to maintain multiple file number/offsets?
< sipa>
how so?
< dcousens>
per block
< sipa>
oh! a block is always in only exactly one file
< sipa>
continuously
< dcousens>
with a possibility to flow into the next file, right?
< sipa>
how do you mean 'flow into' ?
< dcousens>
half in blk001.dat, half in blk002.dat
< sipa>
no
< sipa>
always continuously in one file
< sipa>
if file N can't fit it, it goes into file N+!
< sipa>
which is why the block files aren't exactly the same size :)
< dcousens>
ok! easy
< dcousens>
but the block files themselves could contain complete garbage data mixed with actual data
< dcousens>
conceptually
< sipa>
yes
< sipa>
though that too is something we try to prevent :)
< dcousens>
well, crap. Haha, I'll let you know how I go, but, considered I got 360k blocks in... might be easier to just trash the ?garbage? blocks and see how it goes after
< sipa>
what is the problem?
< dcousens>
"(having troubles with a parser, it makes it ~365k blocks in before blowing up with the magic header being wrong)"
< sipa>
well scan for the first valid header :)
< dcousens>
but the parser makes the assumption the data is contigous with no gaps (aka, no garbage)
< dcousens>
sipa: might do, just wasn't aware that was necessary
< sipa>
i think it's very unlikely to happen in 0.11 or later
< sipa>
would require block index corruption
< dcousens>
is a fresh build and IBD from master
< sipa>
weird
< sipa>
maybe i'm wrong
< sipa>
maybe your block.data is just actually corrupted? :)
< dcousens>
maybe I am, like I said, trying a re-index now
< dcousens>
see how bitcoind goes, if it gets past it
< dcousens>
well
< dcousens>
i know exactly where its blowing up, so I can hex dump and have a looksy :)
< dcousens>
thankfully the parser only takes 23s to run
< sipa>
that's impressive
< sipa>
wait
< dcousens>
no script validation, obviously
< sipa>
you parse 2 Gbyte per second?
< dcousens>
that sounds a bit majestic doesn't it
< dcousens>
and not right...
< dcousens>
but, its a full parse. I've got a 100MB buffer and the disk IO is going hard
< dcousens>
its on AWS
< dcousens>
so reading 100MB at a time, lazy loads the block in terms of what I want out of it (in this case, for a script index)
< sipa>
well how else do you parse 50 Gbyte of block data in 23s?
< sipa>
oh, you don't actually read the block data, just the headers?
< dcousens>
sipa: technically I do though, like, I'm not doing an ftell...
< sipa>
not following
< dcousens>
in terms of IO...
< dcousens>
sipa: 2GB seems a bit fast for what IO should be, AFAIK the AWS SSD's are only about 600mb/s
< phantomcircuit>
the python data node i wrote in ~2010 had a super optimized sqlite3 db and even then it was comically slow
< phantomcircuit>
even with a proper db like postgres it was pretty horrendously slow
< dcousens>
jgarzik: even with prepared statements etc?
< jgarzik>
dcousens, yes
< gmaxwell>
it's not really that sqllite is "slow", it's pretty fast for what it does; it's just that dedicated transactional key value stores are really quite fast.
< gmaxwell>
for me bitcoin core sync runs at around 8000tx/s (with secp256k1), even though every tx is involving a non-cachable missing record check for bip30 (in addition to all the cached stuff)... thats really quite remarkable IMO.
< GitHub101>
[bitcoin] gmaxwell closed pull request #6991: Minor: Clarify 'fee' field in fundrawtransaction help text (master...clarify-fundrawtransaction-help) https://github.com/bitcoin/bitcoin/pull/6991
< GitHub34>
[bitcoin] laanwj closed pull request #7021: add bip65 tests to rpc-tests.sh -extended (in 0.11 branch) (0.11...11rpcfixups) https://github.com/bitcoin/bitcoin/pull/7021
< GitHub193>
bitcoin/0.11 9730051 Alex Morcos: add bip65 tests to rpc-tests.sh -extended
< GitHub193>
bitcoin/0.11 595c8d6 Wladimir J. van der Laan: Merge pull request #7021...
< gmaxwell>
bleh, so it looks like my gbt latency measurements going up coincide not with mempool flooding attacks but with updating to master. Now I need to figure out what hurt the performance so much.
< wumpus>
on the positive side, updating to master is easy to reproduce
< gmaxwell>
well I need to figure out what I was running before! :)
< wumpus>
git reflog?
< gmaxwell>
yea, I'll try (tomorrow, late here)
< wumpus>
no problem, there's always tons of metadata :p
< wumpus>
so it's going to be either a) the switch to libevent's http server, there may be some foul interaction with longpoll b) one of the mempool changes
< wumpus>
there's also c) something other, contending the cs_main lock .. may even be related to the pruning-related changes. Ok, conclusion: no clue.
< gmaxwell>
your thinking is already much more constructive than mine, I was stuck at "oh no" :)
< gmaxwell>
the GBT latency doesn't LP. it's p2pool timing how long it takes it to make a gbt call on its own. The delay is constant, so I doubt contention unless its really hot.
< gmaxwell>
if it's libevent it might just be some silly tunable.
< wumpus>
well it may even be just some silly mistake
< wumpus>
e.g. creating an event but not triggering it
< gmaxwell>
hm. I don't see the time when I moved back to test 0.11.2rc on this graph. Interesting.
< wumpus>
btw using just seccomp-bpf it's not possible to limit the scope of opening files, seccomp scripts cannot dereference syscall argument pointers, it's limited to checks based on the numbers themselves - e.g. disallow PROT_EXEC on mprotect
< wumpus>
so if you allow open, it can open any file on the file system
< gmaxwell>
oh crud, thats .. lame!
< gmaxwell>
well of course selinux and such can control what files you can open.
< wumpus>
it's possible to get around it with a dedicated process that traps some syscalls and then does the syscall for the restricted thread, but that's a lot more complexity
< wumpus>
all those require privileged operations or changes to the OS
< wumpus>
seccomp is IIRC the only one that can run unprivileged, chrome has a set-uid helper to do some jailing operations, but I don't like that
< gmaxwell>
Right, selinux requires privledge setup to establish the rules on the system (though after that the tags/contexts can be applied from userspace); I don't know how the apparmor stuff works.
< wumpus>
apparmor works the same, it is a global config file per application
< gmaxwell>
Hm I would have expected the namespaces thing to have some of the problems chroot has that makes you need privs to use it.
< gmaxwell>
oh well even if we can't limit file scope at least that easily done with selinux/apparmor, and we could limit other things.
< wumpus>
the advantage is that it can be set up externally, on the other hand that relies on specific files for the OS
< wumpus>
nice about seccomp is that it can be set on a per-thread level. Although that doesn't help as much as long as signing and key storage happens within the main process :-)
< gmaxwell>
oh I don't think I realized seccomp could be thread scoped.
< wumpus>
the idea is to set up a global seccomp filter as soon as possible before any threads are created
< gmaxwell>
though use of callbacks and event signaling would then make you have to be careful about what thread things actually excuted in.
< wumpus>
but in a thread it can be restricted further
< wumpus>
the security mesaure that amounst to is block all the exits but what people want to steal isn't outside, it's in the same memory space
< gmaxwell>
e.g. I'd say the network thread could lose all file IO. ... except I'm not sure that it can.
< gmaxwell>
sure if you lose control of control flow you've lost, but if the attacker can just call some libc functions, taking away file IO and such can be pretty limiting.
< wumpus>
so I think we need seccomp lockdown of the main process (at least the one that does P2P networking) + key storage and signing in a different process and memory space
< wumpus>
P2P networking would never have to see the passphrase or decryption key, for example
< wumpus>
"sure if you lose control of control flow you've lost" the whole idea is to mitigate that, that if you have control flow, you can only keep up doing what the thread was doing anyway because nothing else is allowed :-)
< GitHub45>
[bitcoin] jonasschnelli opened pull request #7025: [Qt] refactor and optimize proxy settings behavior (master...2015/11/qt_settingsvalidation) https://github.com/bitcoin/bitcoin/pull/7025
< gmaxwell>
yea, I want to do signing in another process in any case for a multitude of reasons. Including being able to keep passphrases out of ram... being able to use HSMs, etc.
< wumpus>
right
< wumpus>
and my point is that I think that's a more useful step, given concrete thread models, than seccomp right now
< gmaxwell>
annoyingly though, if compromising the threads that are doing network deseralization can be escaped to take over the user account, the likely they can compromise the signing process too.
< gmaxwell>
Ah. Fair enough.
< wumpus>
e.g. the UPnP vulnerability would have allowed getting control of a thread that has network access, no amount of lockdown couldh ave avoided it from leaking secrets from the process memory
< wumpus>
*apart* from having the secrets in a seperate process in the first place
< gmaxwell>
UPNP could also be easily moved into an external program, FWIW.
< wumpus>
that's not my point - UPnP is just an example. The worst case scenario would be an attack through P2P
< gmaxwell>
but thats much less interesting than seperating the secrets from the attack surface, as upnp is just one vector.
< wumpus>
right, better to protect what has to be protected instead
< gmaxwell>
I do think if we decide we'd like to turn upnp back on, it should go in another process and _that_ one we could seccomp to have no access to open(). :)
< gmaxwell>
(even if we are otherwise protecting the signing)
< wumpus>
it's possible to use unprivileged user namespaces, but only on linux kernel 3.18+
< btcdrak>
the message "UpdateTip: 1 of last 100 blocks above version 4" is not correct. It's above 3.
< wumpus>
BTW: if you get "/home/user/bitcoin/src/key.cpp:204: undefined reference to `secp256k1_ecdsa_sign_recoverable'" errors after updating to master you need to clean your git tree (since #6983)
< GitHub134>
bitcoin/master aee22bf Gregory Maxwell: Avoid a compile error on hosts with libevent too old for EVENT_LOG_WARN....
< GitHub134>
bitcoin/master dafefb7 Wladimir J. van der Laan: Merge pull request #7016...
< GitHub72>
[bitcoin] laanwj closed pull request #7016: Avoid a compile error on hosts with libevent too old for EVENT_LOG_WARN. (master...without_EVENT_LOG_WARN) https://github.com/bitcoin/bitcoin/pull/7016
< midnightmagic>
yay!
< wumpus>
user namespaces create a 'fantasy world' for a process in which it is root, but root is mapped (by the namespace) to the user own uid - it does have capabilities such as chroot *within* this user namespace, but it doesn't let it use its elevated privileges outside that. This gives me a headache.
< wumpus>
"Over the years, there have been a lot of features that have been added to the Linux kernel that have been made available only to privileged users because of their potential to confuse set-user-ID-root applications. In general, it becomes safe to allow the root user in a user namespace to use those features because it is impossible, while in a user namespace, to gain more privilege than the root user of a user namespace has"
< wumpus>
not really promising regarding stable API that the example program in "man user_namespaces" is already broken in kernel 4.2.0
< jonasschnelli>
wumpus: sorry for 6900, i could not create a trusty base image. Seems not to work on my debian7 machine. I tried to change the gitian vm builder... but looks like a VitrualBox mkfs issue.
< jonasschnelli>
The only gitian host i have is debian...
< * jonasschnelli>
needs more nativ linux machines...
< phantomcircuit>
wumpus, neat
< sipa>
yay :)
< phantomcircuit>
sipa, how long have you been working on libsecp256k1?
< sipa>
i started somewhere in 2013, i think
< wumpus>
jonasschnelli: doesn't strictly need to be native - w/ lxc you can also use VMs, I think it's even possible to do nested kvm on recent CPUs
< btcdrak>
sipa: wow, that is like a life's work
< jl2012>
Sorry if this is a stupid question. Is there any contingency plan if libsecp256k1 turns out to be a softfork or hardfork?
< instagibbs>
or both? ;)
< sipa>
if there is any difference with openssl, it is a hard fork
< sipa>
because you can negate the outcome of checksig
< sipa>
so any strictening of rules can be turned into widening, in theory
< sipa>
though at this point i think it's fair to say that due the amount of testing and formal validation that went into libsecp256k1, that is more likely to be a bug in openssl then
< sipa>
also, it may not be exploitable... the BN_sqr bug we found in OpenSSL (thanks to libsecp256k1's unit tests that compared with OpenSSL) was technically a hard fork for bitcoin when it got foxed, but an unexploitable one as far as we know
< sipa>
*fixed
< sipa>
so i guess it would depend on what sort of consensus change it would be
< sipa>
and if it's before 0.12 is released, we can always just revert it :)
< instagibbs>
sipa, is there a summary of the review/testing process for it?
< sipa>
instagibbs: not really, but i agree we need that
< sipa>
instagibbs: several types of tests are not yet merged (though they pass fine), including sage scripts to algebraically verify the group formulas, and an alterrnative build that results in a very small size curve so we can do high-level exhaustive testing
< sipa>
those should go in before 0.12, as well as some document describing all the various methods
< GitHub162>
[bitcoin] MarcoFalke opened pull request #7028: [doc] qa: Move README.md and update -help (master...MarcoFalke-2015-qaReadme) https://github.com/bitcoin/bitcoin/pull/7028
< GitHub151>
[bitcoin] sdaftuar opened pull request #7029: Remove unmaintained example test script_test.py (master...script-test-cleanup) https://github.com/bitcoin/bitcoin/pull/7029
< jgarzik>
Great to see libsecp256k1 go in
< btcdrak>
and openssl to be shown the door
< GitHub155>
[bitcoin] MarcoFalke opened pull request #7030: [contrib] Delete test-patches and move testgen to devtools (master...MarcoFalke-2015-contribC) https://github.com/bitcoin/bitcoin/pull/7030
< wumpus>
jonasschnelli: don't really understand
< jonasschnelli>
wumpus: The 6900 issues?
< wumpus>
yes
< jonasschnelli>
seems something with boost
< wumpus>
I thought the point of gitian was that the results are the same for everyone
< wumpus>
you're also using kvm? or lxc?
< jonasschnelli>
ha... right.. maybe i do it slightly different..
< jonasschnelli>
wumpus: kvm,... not i try to build 0340ffb
< jonasschnelli>
not/now
< wumpus>
the non-working boost sleep implementation error usually means it cannot link against boost at all
< wumpus>
unfortunately it's not really easy to get the config.log in the general case from the gitian builder
< wumpus>
linux seems still to be building properly here - although it has the dependencies cached, could try removing the cache and starting over from the beginning but that's going to take a long time
< jonasschnelli>
morcos: I have to admit, that i haven't fully reviewed #6134, my focus was the GUI part.
< morcos>
jonasschnelli: i just wanted to make sure i understood your comment about never returning -1
< jonasschnelli>
I think for an user it's strange that he might get the same fee for targeting conf. withing the next 25blocks and the next 18blocks.
< jonasschnelli>
morcos: I was calling estimatefee instead of estimateaproxfee
< morcos>
part of the point of the PR was to make it so you got -1 a lot less often. but i didn't change what estimatefee returns, i just made a new function estimateapproxfee which is smarter
< morcos>
perhaps i should call it estimatesmartfee instead
< morcos>
that one will still return -1, but only at the very startup when there is not enough data period (and you don't have an old estimates file, although thats maybe dangerous for its own reasons)
< jonasschnelli>
I think the name is okay. I just wasn't really concentrated while reviewing and mixed up the extimateaproxfee and estimatefee
< morcos>
ok
< morcos>
as for same fee for different confirmations
< jonasschnelli>
UI: why can't you target for 1 block?
< jonasschnelli>
(conf. withing next block)
< jonasschnelli>
*within
< morcos>
i'm not really sure there is anything to be done about that. its just as accurate an answer as you can get from the data
< morcos>
how do you mean, you can target for 1 block
< jonasschnelli>
The slider i got was from block 25 to block 2.
< jonasschnelli>
So, as enduser i would whish i could select 1 block.
< morcos>
ok, so that is the UI change i wanted fee back from
< morcos>
if you look at the right end of the slider
< morcos>
position 1 and position 2, both say 2 blocks
< jonasschnelli>
Right. I saw that. Is that because 1 block is just very hard (impossible to estimate)?
< morcos>
so you are actually trying to get an estiamte for 1 block, but estimatefee returns -1. so estimateapproxfee looks at 2 blocks, gives you an answer there, and lets you know that the answer is for 2, so you're not expecting to get confirmed in 1 with that fee
< jonasschnelli>
* ... hard (impossible) to estimate?
< morcos>
yes, thats correct, there is not enough data to indicate that any fee will have very high likelihood of being confirmed in 1 block
< morcos>
this is primarily because i require 95% (used to be 85%) of txs with that fee or higher to be confirmed within the desired number of blocks, and often miners only recalculate their block every minute
< morcos>
so that puts a ceiling of about 90% on the chance you will be confirmed in the next blcok
< morcos>
my prior version of the change had the slider snap back to position 2, but that seemed like i did not implement that very well. so the simpler change is this, but i'm happy for suggestions on how to improve it from a UI standpoint
< morcos>
what i'm trying to communicate is we can't give you an answer for 1 block, the best i can do is give you an answer for 2 blocks and that answer is X
< jonasschnelli>
hm... hmm.. somehow it feels that the user must be able to select 1 block... i know, it's not possible to estimate... :)
< jonasschnelli>
we could also go the way other wallets do: "confirmation speed" : "slow" = ~8b | "avg" ~4b | "fast" ~2b.
< jonasschnelli>
we just use the words "slow", "average", "fest"
< jonasschnelli>
*"fast"
< morcos>
i do think the ability to choose any # is overkill
< morcos>
but maybe thats too big a change now for 0.12?
< jonasschnelli>
I mean, it's nitpicking. The change loos good to me (need to test more in detail).
< jonasschnelli>
*looks
< jonasschnelli>
what do you think by aggregating block/fees.
< morcos>
ok... what do you think about changing estimateapproxfee to estimatesmartfee. i think i like that name better
< morcos>
better holds to how i think that function will evolve in the future
< jonasschnelli>
yeah. Smart* sounds always good. I also like estimatesmartfee better.
< morcos>
aggregating block/fees?
< jonasschnelli>
the thing is, if you move the slider, you might get the same fee between block 25 and block 18 (was the case in my test),...
< jonasschnelli>
we could just stop at block 18 (as min value, to very left).
< morcos>
hmm.. the problem is its not really possible to know any 2 targets would give you the same number until you ask
< jonasschnelli>
we could run from 2-25, then aggregate values, invalidate (this cache) after a new block comes in (or similar)
< jonasschnelli>
but, yeah. sounds after some lines of code.
< jonasschnelli>
could also be done later.
< jonasschnelli>
I mean the PR should not be hold back because of the UI!
< morcos>
yeah i think after we get 0.12. we should talk about how to improve it
< morcos>
maybe do something like only store results for 1,2,4,8,16 (and then possibly add longer time horizons: 32,64,128,256?)
< morcos>
and then thats the only place the slider stops
< morcos>
perhaps they will sometimes give the same value, but a lot less often
< jonasschnelli>
yes. This could make sense.
< jonasschnelli>
IMO we should keep the UI as it is for #6134 and focus on get it in.
< jonasschnelli>
What is the reason to have two estimatefee rpc calls. Keep the API?
< morcos>
ok agreed. especially as i am out of town for 10 days starting on friday!
< morcos>
yeah i think fee estimation still has some evolving to do
< jonasschnelli>
Take your laptop with you. :)
< morcos>
so i'd like to have estimatesmartfee be the new API, that will evolve with the functionality
< morcos>
but for the meantime keep the old API consistent until we have a sense of where the smart version will end up
< jonasschnelli>
will it only be estimation? Or would a RPC call "smartfee estiamte" make more sense?
< morcos>
i didn't understand
< jonasschnelli>
*estimate
< jonasschnelli>
I'm just thinking loud. Maybe other smartfee interaction would be possible in future, setting some parameters or similar.
< morcos>
oh, like you could configure state within the estimator, like smartfee timehorizon 1008
< jonasschnelli>
Then we could collect everything in the smartfee rpc call.
< morcos>
well i think maybe i should just call it smartfee and smartpriority ?
< jonasschnelli>
something like "smartfee estimate" and "smartfee set timehorizon 100"
< morcos>
i'm not sure we'll want to set state like that, i think it would be safer to only be able to configure parameters on startup
< jonasschnelli>
Hm.. no. better keep it "estimatesmartfee".
< morcos>
ok... estimatesmartfee it is. i'll rebase at some point with just the name change...
< jonasschnelli>
We could still have something like "estiamtesmartfee settimehorzon X"
< jonasschnelli>
I'll try to do a full test after your rebase.
< morcos>
ok
< jonasschnelli>
Keep the UI stuff as it is. We can polish that later.
< morcos>
yikes, so many commits, i'm just going to make one final commit which changes the names i think...
< jonasschnelli>
Ok.
< jonasschnelli>
morcos: one thing: I think we sould remove ""\nWARNING: This interface is unstable and may disappear or change!\n""
< morcos>
oh why? i wanted to have that in there so people didn't write code depending on estimatesmartfee
< morcos>
i'm keeping the estimatefee interface the same
< morcos>
and for the most part they can design their own smart algorithm by using estimatefee directly
< morcos>
but i don't want to tie down the interface to the built-in smart algorithm now
< morcos>
i almost didn't expose it as rpc calls, but its nice for testing
< jonasschnelli>
Well, i think if we pack that into a release, we need to consider that people write code depending on that... new features in a 0. version are not meant to habe stable API IMO
< jonasschnelli>
*have
< morcos>
well thats the point of the warning, so they don't write code or they're aware. i think that was wumpus' recommendation
< jonasschnelli>
But no strong opinion on that.
< jonasschnelli>
Okay. Then lets keep this.
< morcos>
ok thx
< jonasschnelli>
morcos: nice code/PR by the way. You guys always come directly with a rpc test and a bunch of unittests!