< phantomcircuit>
CodeShark, definitely cant brute force all the 256 byte combinations :)
< CodeShark>
is the goal simply to find valid transactions that are exactly 256 bytes long?
< BlueMatt>
phantomcircuit: do you need a faster box to run that on?
< CodeShark>
or nvm, I don't really know what you're trying to do :p
< phantomcircuit>
CodeShark, no the goal is to find interesting transactions
< phantomcircuit>
valid or otherwise
< phantomcircuit>
BlueMatt, i dont think it would matter much, it's been running for about two days and hasn't found a new .... wait
< phantomcircuit>
damn it i screwed up the afl-fuzz setup!
< * phantomcircuit>
grubles
< CodeShark>
by "valid" I'm just talking in terms of parser logic - not the chainstate and crypto validation stuff :p
< BlueMatt>
phantomcircuit: if you send me the vm image I can run it on my workstation.....
< phantomcircuit>
hmm actually maybe not
< phantomcircuit>
looks like i didn't screw it up
< phantomcircuit>
CodeShark, i have 1 testcase which fails the parser logic, but because of how afl-fuzz works it's only th eone
< phantomcircuit>
gmaxwell, there seems to be an issue with the frequency of wallet flushes and the pruning logic
< phantomcircuit>
the pruning logic can prune blocks where the wallet hasn't flushed indicating it's sync'd to them
< phantomcircuit>
BlueMatt, you should fix! :P
< BlueMatt>
huh?
< gmaxwell>
hahah
< gmaxwell>
phantomcircuit: yea that sounds bad. :P
< phantomcircuit>
i would bet the same thing can happen with the block index, but i'd need to double check on that
< CodeShark>
it would really be nice to put all the header logic into its own unit
< CodeShark>
separate issue...but just thought I'd bring it up :p
< CodeShark>
then other parts of the app can sync to the headers in different ways
< CodeShark>
and the header index can carry additional state information
< CodeShark>
(i.e. what rules to enforce for the particular block :) )
< CodeShark>
we could also shave off 80 bytes of data from getblock if we always use getheader to grab the header
< CodeShark>
not that 80 bytes is that much when put into context of downloading the entire block :p
< CodeShark>
but we could have more sophisticated queries for partial blocks
< CodeShark>
I'm very much not satisfied with the filteredblock mechanism we currently have ;)
< CodeShark>
even pruned nodes could serve useful data if the query structure were better
< CodeShark>
in retrospect, perhaps Satoshi should have tried to implement SPV first :p
< gmaxwell>
he did.
< gmaxwell>
there was a begin a it in bitcoin core that was long since stripped out
< CodeShark>
begin a it?
< gmaxwell>
at
< CodeShark>
I'm saying perhaps the header sync should have been the first consensus code written
< gmaxwell>
today it will be much easier to do, but long ago the architecture was probably a bit too alien.
< gmaxwell>
CodeShark: you have some hindsight benefit of how we think about the architecture of the system in modern times. :)
< CodeShark>
yes, I said "in retrospect"
< CodeShark>
in terms of what we can do now, I think if we really want to build a consensus library we should think in terms of isolating header sync into a standalone unit
< CodeShark>
then we can build whatever kind of node on top of that
< CodeShark>
whether it be full validation, pruned or not, SPV, etc...
< CodeShark>
it will also help in avoiding unintended forks and other such issues in the future as we can explicitly know which rules can and cannot be enforced by a given node
< CodeShark>
there's a whole spectrum here
< CodeShark>
from "check absolutely everything" to "check nothing"
< gmaxwell>
the structure of bitcoin core today is much more agreeable to that than it was in the past.
< CodeShark>
yes, sipa did a good job with the headers-first sync
< CodeShark>
so it's going in the right direction
< CodeShark>
I did some outlining of everything that happens in ProcessNewBlock...there's a lot of redundancy and plenty of things that can be cleaned up
< CodeShark>
ContextualCheckBlockHeader gets called twice
< CodeShark>
it would be nice to just make all the state info checked in that part of the header index itself
< CodeShark>
ultimately, I'd even like the version bits logic itself to be directly part of the header index - so instead of doing something like Rules rules GetRulesFromBlockIndex(pblockIndex); you can just do blockHeader.GetRules();
< CodeShark>
or headerIndex(block_hash).getRules();
< CodeShark>
or whatever
< CodeShark>
something pretty like that :)
< CodeShark>
then the rest of the block checking stuff can have conditionals like if(rules.contains(BIP66)) { }
< CodeShark>
and from that point on we can easily isolate ANY changes to consensus checks and test them independently
< CodeShark>
even alt coins could be built that can contribute useful ideas back to Bitcoin ;)
< CodeShark>
rather than having to constrain our refactoring efforts to avoid pushback from people who long ago forked Bitcoin Core :p
< sipa>
phantomcircuit: no, that can't happen with the block index afaik, as the flushing and pruning logic are aware of each other
< sipa>
phantomcircuit: but for the wallet... i guess that's possible
< morcos>
phantomcircuit: sipa: yeah wumpus and i went through the block index / pruning logic a week or so ago, and it is a tiny bit scary, but it works. i promised him i'd write it up in an issue, its still on my to do list. we'll check into the wallet
< morcos>
actually i guess for the block index, is more about just commenting the code as to why it works, so no one changes it
< CodeShark>
after the 0.12 release, I propose we work towards breaking out headers sync as the first part of modularizing and encapsulating the consensus code :)
< CodeShark>
we should have a standalone module that can do a headers sync and provide query and subscription services
< CodeShark>
not necessarily even as a standalone library - just a unit that can be built along with a simple demo app that only uses stable dependencies
< CodeShark>
all the protocol rule changes (including versionbits) can naturally be fit into this
< CodeShark>
I'll write up a more detailed document if enough people seem sufficiently interested to make this worthwhile pursuing and encourage anyone to join in
< sipa>
CodeShark: synchronization is very different from consensus :)
< CodeShark>
this obviously needs to be done in coordination with all our other refactoring efforts
< CodeShark>
sipa, I should clarify...
< CodeShark>
I meant block header consensus - this example I'm sharing has no networking code
< CodeShark>
it allows you to insert block headers (that can come from any source) and it can validate PoW, timestamp, and version
< CodeShark>
and can allow clients to subscribe to signals for events...and we can plug in modules for doing more verification with actual block data
< CodeShark>
then we'll have total flexibility - we can do headers-only sync...or we can do full validation...or anything in between
< sipa>
CodeShark: i've suggested something similar, by moving full validation to a separate module with its own state
< randy-waterhouse>
a hybrid node
< sipa>
which uses the main signals to trigger full validation
< CodeShark>
yeah, I think it's the way to go
< sipa>
i think my point is that that interface already exists
< sipa>
and we should reuse it
< CodeShark>
can I see it?
< sipa>
but rather than extract header validation from it
< sipa>
move full validation out of it
< sipa>
eh, look at main.h, there is a signals interface
< sipa>
though i'd start with the wallet
< sipa>
allow it to have its own "active block", which can be different from that of main
< sipa>
ot only subscribes to updatetip events, and then requests the block data asynchronously to sync
< sipa>
full validation can work the same way, but needs a slightly more featureful interface, where it can mark a block as invalid
< sipa>
CodeShark: also, i disagree with moving more logic into basic data structure classes
< sipa>
we've been moving the opposite direction mostly, because it really interferes with modularization
< CodeShark>
well, it doesn't have to be implemented in the interface classes - we can either use templates or inheritance or something ;)
< sipa>
bleh, even uglier
< CodeShark>
or delegation
< CodeShark>
why uglier?
< sipa>
let me give an example to elaborate, so we're talking about the same thing
< sipa>
imagine ctransaction had a sign function
< sipa>
sounds awesome, very useful and elegant
< CodeShark>
oh, that's not what I'm saying at all!!!
< CodeShark>
I actually think that's hideous
< sipa>
you were saying the block should have a function to ask what bips apply to it
< CodeShark>
CTransaction should just contain the transaction data in a way that's easily accessible by the app and provide serialization
< sipa>
i think that's very similar
< sipa>
ok, we agree there
< CodeShark>
regarding asking what bips apply, we need to maintain an index for this...so it seems like it would be convenient to have the same entrypoints we use to grab other block header info...but there are several alternatives to this
< CodeShark>
and in fact, it makes sense to NOT build this logic into the core header consensus stuff
< CodeShark>
at least not directly
< sipa>
good :)
< CodeShark>
I'd even leave room for flexibility on PoW and timestamp verification :)
< CodeShark>
and if we want to go further, we can even make the header structure itself abstract
< CodeShark>
the key structural element at this level is the notion of a block header tree
< CodeShark>
which branch is "active" and the rules we apply to connect new headers needn't be specified at this level
< CodeShark>
this stuff can be provided by a number of different mechanisms
< sipa>
it makes more sense to do a writeup together with some minimal idea for code changes then talk here
< sipa>
irc is good for discussion, but not really for monologue
< CodeShark>
yeah :)
< CodeShark>
so google docs?
< sipa>
or issue
< sipa>
or a branch
< CodeShark>
I'd like to approach this from both directions, top-down and bottom up. I'd like to consider high level architectural stuff to set some goals...and then consider the practical implementation issues...and then put together a plan that can get us to our goals in incremental steps that are palatable
< sipa>
so i prefer a high-level plan (without code) to show the intent, and then low-level changes that help get therr
< CodeShark>
yes, absolutely agreed - the high-level plan should be much more about diagrams and ideas than about code
< CodeShark>
we'll inevitably need to make some design decisions along the way that are largely determined by practical considerations...but our ultimate objectives should be clear
< CodeShark>
moreover, we don't need to spell out all the individual steps up front - it is sufficient that we specify a few initial steps that, regardless of what we decide later, move us in the right direction