< fanquake>
wumpus no pressure on your new years eve!
< wumpus>
hehe for some reason I never saw the copyright year bumping as particularly pressuring, but we don't like a flood of PRs so I agree it's good to have it over with :)
< wumpus>
don't they need t run some script to update the copyright in the source code as well
< fanquake>
Yea, Marco has a another PR open that's doing that
< fanquake>
We could probably combine the two, but I thought given they are both open already we can just get it over with hah
< cjd>
This is funny, 4 years ago bitcoin could only possibly work on little endian (probably only on i386/amd64) and cjdns had nice endian swapping macros and all of that
< cjd>
now I see endian swapping macros in bitcoin and they look almost exactly like my old code
< cjd>
but now, I'm nolonger writing code with endian swapping, I consider all processors that matter to be EL and I'm holding my breath for the last of the BEs to die out
< gmaxwell>
cjd: our working correctly on BE has very little to do with caring if the software works correctly on BE, and a lot more to do with code that doesn't is almost always executing undefined behavior, and is potentially vulnerable to the compiler being especially creative as a result.
< cjd>
you mean aliasing ?
< gmaxwell>
Yes, thats one sort of undefined behavior that will cause you endianness handling issues (and which, in fact, will cause compilers to do things you probably didn't expect, in practice not just theory)
< gmaxwell>
endian fragile code also usually mishandles alignment.
< gmaxwell>
(esp when most of the developers are on x86, which usually doesn't crash on alignment mistakes... unless the alignment mistake crosses a page boundary, and the compiler manges to use one of the aligned simd loads...)
< cjd>
I've used alignment assertions in the past because I generally would rather everything be aligned than not know
< cjd>
but if you're unpacking tight structures like transactions, swapping byte order is no real extra overhead since you have to be quite careful with them anyway
< cjd>
*being prepared to swap them
< gmaxwell>
it's also no overhead because the compiler will totally eliminate the swaps where they aren't needed.
< cjd>
runtime overhead, I was thinking development overhead (which is my main concern)
< cjd>
easy to make things have no runtime overhead in most procs, specify everything as EL
< sipa>
thankfully we don't (need to) touch the serialization code very often
< gmaxwell>
(also, on lots of processors, though not x86, byteswapping is essentially free because it just gets turned into a flag on load instructions)
< gmaxwell>
e.g. you wire encode little endian, on x86 the macros just vanish entirely, and on (say) BE-power9 it turns into the byteswapping version of the relavant load instruction, and the result is as fast as if only one way existed. :)
< gmaxwell>
as far as the code goes--- you've got to have something there where you are reading from char into ints. Just casting is not correct (prohibited by aliasing, alignment, or both).
< cjd>
I tend not to cast pointers that often, though I do a fair amount of memcpy
< gmaxwell>
right, well, if you have to call memcpy you can just call something else instead that handles byteorder if required.
< gmaxwell>
uhh, so you know that the structs layout in memory is totally not portable, right? :P
< gmaxwell>
not just in terms of endianness.
< cjd>
yes, and I also know that linux will run into problems before I do :P
< cjd>
I don't use bitfields in structs I'm planning to write on the wire, they do
< cjd>
I also use explicit pad fields and generally static assert the size to be reasonably assured that the compiler didn't imagine some place to add padding
< gmaxwell>
Linux isn't written in C. in particualr it sticks in all sorts of specialized directives at the compiler to fix the behavior, and then only works on a subset of systems (which is fine, since its an OS its naturally restricted there. :P )
< sipa>
with enough safeguards against all possible choices a compiler/architecture could make, i'm sure you can make this reasonably safe
< sipa>
it seems much more of a worry than just using layout-neutral code though :)
< gmaxwell>
welp, when some clever compiler operations turns that code into something that gives remote root to the world, you'll have only yourself to blame. :P
< cjd>
ehh
< sipa>
it's probably getting a bit offtopic here
< cjd>
if (programmer_used_ub()) { insert_backdoor_code(); } I don't think the compiler authors are completely off the hook
< cjd>
sorry
< cjd>
happy newyear btw
< gmaxwell>
In any case, new code under the logic you're using wouldn't be accepted to this project without a really strong justification. Other projects might do different things, sounds like a problem for their channels.
< cjd>
Yeah, when I'm trying to get something accepted I follow whatever rules the project uses
< wumpus>
IBM power arch can use big-endian and apparently some linux distros use it with that, I think it's part of basic correctness/abstraction to have software work independent of CPU endianness
< midnightmagic>
+1 endian-independent design..
< gmaxwell>
I wish someone made a (could be simulated only) arch that made every free decision opposite. Endianness, stack growth direction, calling conventions, etc. I've found more than a few all-platform bugs from testing code on BE simply because differences in the execution enviroment turned a usually invisible bug into an instant crasher.
< wumpus>
although I agree there are some decisions, and choices with regard to architecture that simply die out, there's no need to support any architecture, real and imagined since the 60s, say, no need to handle the case where int is 16 bit
< gmaxwell>
an "adversarial C compiler" would be fun, ... tries to behave in a way most likely to make everything fail, subject to being strictly standards conforming.
< wumpus>
or char is two bytes...
< wumpus>
maybe big endian will be one of those in the future but I don't think we're there
< gmaxwell>
wumpus: I agree with you, though I have written code for char-is-two bytes platforms.
< cjd>
wumpus: that's my thinking when I do projects these days
< gmaxwell>
(TI VLIW DSPs)
< midnightmagic>
Bitcoin can't run on an Amiga. :-P
< cjd>
gmaxwell: reminds me of libdisalloc
< wumpus>
cjd: I don't start projects in C anymore anyway :)
< midnightmagic>
:-(
< cjd>
good move, though I have not seen a really good replacement when speed is a high priority
< wumpus>
(not that that avoids the endian question, but, say, rust has all common types fixed-size, so it avoids the 'is this type one or two bytes' question at least)
< wumpus>
or 'is char signed', eeh I made some terrible mistakes with that
< gmaxwell>
wumpus: I assume there are also standard types in rust for "the fastest thing that is at least 16 bits"?
< wumpus>
gmaxwell: no, there's i16, u16, nothing else IIRC
< sipa>
C++17 introduces the 'byte' type, which allows accessing the representation on memory of other types, without being an int type
< wumpus>
gmaxwell: though, thinking of it, wouldn't be surprised if it exists in some crate
< gmaxwell>
sipa: it's like char but without the char signness baggage?
< wumpus>
gmaxwell: it's just not the common way of doing things and I like that
< sipa>
gmaxwell: it's basically an 8-bit vector without integer aspects
< sipa>
so no arithmetic, only bitwise operations
< midnightmagic>
I'm torn between the nature of the safety decision against using C and/or assembly as a learning environment, and the problematic results of C projects done badly. All I can say is I think it's vewry unfortunate that people are moving away from the bare metal level in general, as it generates a type of ignorance in the average which makes us in general more vulnerable to problems at the
< midnightmagic>
machine level. :(
< gmaxwell>
sipa: but its allowed to alias like char?
< gmaxwell>
e.g. can you implement memcpy using it?
< cjd>
I want to love rust but I'm afraid that too much logic goes into the compiler itself and it's not general purpose enough... so my concern is that some wizard is going to come down from the hill and make a language with 7 builtin keywords and the ability to DSL just about anything...
< sipa>
yup
< sipa>
gmaxwell: yup, byte pointers can alias any other pointer
< gmaxwell>
sipa: sounds reasonable.
< wumpus>
midnightmagic: I like assambly (though I don't use it a lot in practice because I tend to write platform independent software!), I don't really like C
< wumpus>
all the UB pitfalls are not necessary in a langage and harmful
< midnightmagic>
wumpus: I have a secret love for C which as far as I can tell has never dissipated even when I stopped enjoying owning most computers due to backdooring issues. But I don't think people who prefer C or something like it are fetishists-- not quite yet anyway.
< midnightmagic>
oh well
< gmaxwell>
Well the specific way C UB works is not so lovely. Rust has things which are defined to be an error and only caught in debug builds, but have well defined semantics otherwise (even though your code is still technically incorrect if does them).
< wumpus>
'baggage' is kind of the right word, anyhow, so many exceptions and irregularities for history's sake
< sipa>
i like C as a lowest common denominator... if something has to work everywhere, C will do the job
< cjd>
problem w/ C is as gmax pointed out, UB is something that people use every day and the compiler is technically allowed to do anything.. in the words of HST: "In a closed society where everybody's guilty, the only crime is getting caught."
< wumpus>
and this is not speaking as a programming newbie but someone who made every mistake that is possible, and still makes them even after so many years
< cjd>
And I still cast pointers and violate aliasing rules, because it's fast
< gmaxwell>
you /really/ should not.
< sipa>
cjd: i'm a bit skeptical that it actually gains you any speed
< gmaxwell>
correctly written code is seldom any slower. (not to mention that only a tiny fraction of code is actually performance relavant)
< sipa>
for example a function uint32_t readLE(const unsigned char* data) { uint32_t ret; memcpy(&ret, data, sizeof(uint32_t)); return letohs(ret); } will basically compile to just reading memory on x86
< wumpus>
cjd: yes, it's not a good situation IMO, people tend to overestimate themselves, 'I can make this violation because I know what I"m doing' and maybe that's true part of the time, but the times it's not make it very dangerous in critical code, and lots of code is critical today
< gmaxwell>
I've found that its more common that convincing the compiler that there isn't any of the _permitted_ kinds of aliasing ends up being important for performance, since it allows vectorization.
< cjd>
agreed on that point
< sipa>
OAOBle32toh, sorry
< wumpus>
also yeah compilers change, violations that could be done more or less safely in the past no longer safe now, you have this running in the same place to keep up with the ocmpiler
< gmaxwell>
wumpus: also they correctly estimate their best, or even just average, capability... which is a massive overestimate of them at their dumbest moment... :)
< cjd>
I like the Rust vision of aliasing since in theory the compiler gets a lot more freedom
< gmaxwell>
The same humans that one in one-hundred morings try to put their pants on inside or or their shirt on backwards... :)
< wumpus>
gmaxwell: yep! this is what makes unsafe {} in rust better than having *everything* unsafe, I guess
< cjd>
we're slowly inching in the direction of everything being proven
< wumpus>
at least the 'I'm doing crazy things here' sections are clearly cordoned off for review
< gmaxwell>
cjd: well rust behavior is good for performance in other ways... you're forced into using iterators for everything to escape constant bounds checks, but then that also gives the compiler more freedom.
< cjd>
very good point
< MarcoFalke>
Should we add a repo to bitcoin-core with fuzz seeds?
< cjd>
I really do want to love rust, I think my biggest complaint is the fact that it's static assertion capability is significantly weaker than C's
< wumpus>
cjd: I hope so re: the "being proven" part, that languages are going to allow specifying high-level invariants that can be checked, and that using this is going to be common sense
< gmaxwell>
cjd: I don't really think there is much in the way of material progress in terms of provability. like I think I would agree that we're moving in the direction where something people actually use is proven in a way that could have avoided problems... but everything? dunno there. It's not like new and trendy languages like go or rust were designed with making proving things about code writeen in
< gmaxwell>
them easier.
< gmaxwell>
right now the only code that is even somewhat easy to prove properties of is code that generally doesn't need proofs.
< wumpus>
gmaxwell: rihgt
< cjd>
Proven code in the traditional sense is difficult/expensive and unlikely to gain a lot of traction, but even Javascript is "proven" not to segfault
< gmaxwell>
thats a really low bar, however.
< cjd>
I expect proof to creep in from the side of proving what code cannot do rather than proving what it will do
< gmaxwell>
and consider, JS code for bitcoin has a had flaws that cost users funds many many many more times than code written in languages that aren't segfault free. :P
< gmaxwell>
of course, getting rid of one or more whole classes of bugs is a win. More time to spend on other things.
< wumpus>
I mean that's already the case with rust, by using only safe code (and assuming dependent crates don't violate assumptions) you're essentialy proving you can't have certain classes of bugs
< cjd>
Bitcoin is a unique project, in that it's a piece of code which is typically not exploited to get access to someting else, it's exploited for it's own purpose
< gmaxwell>
But I think that e.g. JS hurts program safty in ways that more than make up for the lack of segfaults.
< gmaxwell>
you can make C code segfault free too-- just run it in a sandbox that halts cleanly when native execution would have segfaulted.
< gmaxwell>
e.g. the nacl sandbox essentially gives code written in whatever langauge the same 'provable' properties as JS.
< gmaxwell>
Yet there seems to be no move to go deploy things that way.
< cjd>
I contend that even the worst languages are valuable because they advance the state of the art, somewhere in some university there is a guy who will no doubt argue that all languages except haskell must be banned, but if he had his way then the industry would still be in the early 90s if not earlier
< gmaxwell>
even though the performance hit is radically lower than any of the popular interperted langauges.
< cjd>
(IMO)
< gmaxwell>
I think a lot of things were better in the 90s. My document readers were actually document readers and not remote spy device execution enviroments for obfscuated non-free software. :P
< wumpus>
cjd: I'd be quite interested in seeing that timeline (given the questionable assumption a programming language ban works in the first place...), everyone using the same language and it happens to be haskell :)
< gmaxwell>
presumably people would just extend haskell until it contained all of C++ and JS. ... :P
< * midnightmagic>
jumps out window
< wumpus>
gmaxwell: yes, it's a mistake to think of progress as linear, even in the area of computing
< gmaxwell>
cjd: also a lot of the computing systems I use today are significantly slower and less responsive than what I used in the 90s.
< wumpus>
yes
< cjd>
There are some people who think that the 50s was the best time in history
< wumpus>
good old time when hardware improvements could still keep up with growth in software bloat :-)
< gmaxwell>
even just desktop applications have degraded in performance, stuff like compositing displays, anti-aliasing, and smooth scrolling have added multiple frames of delay. Which is in fact visible... and thats without getting to the insanity of the web. Every once in a while I dig up a really old system to pull some data off it and end up being surprised at how fast it is.
< gmaxwell>
I don't mean this in some kind of vague 50s-were-the-best I mean by objective criteria that most of us would agree are important.
< cjd>
So stuff like electron is making programming more accessible to a wider number of people, the cost is that things get slow, but that will eventually be solved as well
< gmaxwell>
Like "millseconds to respond to user input in the 99th percentile". Or, "doesn't secretly send your private data to third parties".
< gmaxwell>
I'm sure things will improve again later.
< wumpus>
I'm not as hopeful about the future
< cjd>
Machines get faster, programming gets easier, and typical applications remain just within the range of "tollerable"
< gmaxwell>
For a number of years my main display was an IBM T221, a more than decade (at the time) outdated display, because pretty much from the moment HDTV came out for about 10 years thereafter it became impossible to get a computer display that was higher resolution than 1080p at basically any price. But eventually, 4k+ displays started being made. It just took a long time.
< cjd>
but to throw us back to the age of COM C++ programming would be to remove maybe 80% of the programmers from the programmer pool, so then less software and then less choice
< wumpus>
I'm not convinced that is true, a lot of learning, say, JS is baggage as well
< gmaxwell>
That isn't at all clear to me.
< wumpus>
easy programming languages existed in the 90's as well
< gmaxwell>
some of them were actually pretty effective, like hypercard.
< wumpus>
yes
< gmaxwell>
(in a way that I dont think you can credit JS with being)
< gmaxwell>
(I mean actually effective at allowing less skilled/trained people to create non-trivial useful programs with only moderate effort)
< cjd>
well, sometimes evolution carries us down a fluke path, we have to try a lot of things wrong before we get something right
< wumpus>
JS has so many weird frameworks, different ways to do UI, the HTML UI manipulation isn't *that* intuitive, and with so many different ways a lot of people get confused
< cjd>
JS frameworks are a primordial soup for design patterns
< gmaxwell>
also basically no feedback on why things aren't working right when you mix the soup parts incorrectly.
< gmaxwell>
it does benefit from an enormous amount of code you can copy/paste import, but a lot of it is fairly low quality, undocumented, etc.
< fanquake>
npm install *everything*
< * sipa>
imports the is_integer npm package
< wumpus>
people also overstate the accessibility asspects, it's not that modern UI frameworks, no matter how bloated, really that useful (by default) by someone using a screen reader, braille display, etc, in a way console/text based interfaces were *better* for that
< * midnightmagic>
sadly stares at the smoking crater where sipa was
< gmaxwell>
wumpus: Some of the regression in objective quality I think is because people often dont bother to write down what they did well, and as a result things get replaced by things that are worse but look better.
< wumpus>
I don't think much progress is being made and we're seeing lots of cambrian explosions without follow-up winnowing down phase, so many competing things, ways to do the same
< wumpus>
gmaxwell: yep...
< gmaxwell>
wumpus: I lamented this a lot back when banks moved from terminal based applications to windows UI applications... and visits to the banks started taking 5x longer (and only ever recovered to 2x longer even after years). The gui apps were faster to onboard staff perhaps, but expirenced staff was much slower with them.
< cjd>
That's an interesting observation
< cjd>
Banks are not exactly known for efficiency so one can expect practically anything to happen in their IT systems, even to be written in brainf*ck because a number of tie-wearing men declared that it would be better/faster/safer/higher quality...
< cjd>
But when efficiency is critical, one would expect to see GUI apps with keyboard shortcuts so that there is a smooth learning curve
< wumpus>
gmaxwell: yes, learning curve is not really taken into account, in ways it seems that UI went from a serious study to 'what looks nice and seems intuitive at first glance', with different companies doing different things and zillions of different metaphors
< wumpus>
cjd: you'd expect so
< cjd>
At the end of the day, we probably ought to evaluate programming languages using the anthropological toolkit used for religions because I think they tend to spread similarly
< gmaxwell>
I think there is a general pattern with tools, where first generation tools are barely usable, second generation are radically improved, and third generation try to improve initial impressions or learning curve and ultimately limit the tool.
< cjd>
I suspect the same methodology works for evaluating editors, distros and blockchains
< bitcoin-git>
bitcoin/master 3019ba2 Ben Carman: Making supported operating systems more clear
< bitcoin-git>
bitcoin/master f5a70d1 Wladimir J. van der Laan: Merge #14974: doc: Removing redundant line: "Windows XP not supported"...
< bitcoin-git>
[bitcoin] laanwj closed pull request #14974: doc: Removing redundant line: "Windows XP not supported" (master...release_notes_doc) https://github.com/bitcoin/bitcoin/pull/14974
< cobega>
running bitcoind "walletversion": 169900, after creating a multisig address using "addmultisigaddress" and depositing some coins to this address, using the "listunspent" command doesn't list this address. Is some more efficient way then using "importaddress" (since this will block the bitcoind for a while).
< achow101>
cobega: did you do listunspent with the watch_only option set to true?
< achow101>
you can use importaddress with rescan false, then rescanblockchain from the block height that that address has its first transaction.
< cobega>
moment, will try first listunspent with watch only set to true.
< achow101>
cobega: you can use importmulti too with a timestamp of the block height you first used the address
< achow101>
gwillen: addmultisigaddress adds the redeemScript to your wallet
< achow101>
so you can sign a transaction that spends the multisig without having funds sent to it showing up in your balance
< gwillen>
achow101: ahh so it makes it solvable
< gwillen>
that makes sense, thanks
< sipa>
gwillen: createmultisig is the utility non-wallet version of the RapC, to just let you compute the address
< sipa>
addmultisigaddress is the wallet importing version
< cobega>
using importmulti, will I have to set rescan to false, when I used the option timestamp of it, or would I need to set rescan to true anyways?
< achow101>
cobega: the timestamp indicates from when to start rescanning. it basically combines importaddress and rescanblockchain
< cobega>
Understand :) But should I set rescan to true or false when using the timestamp parameter?
< achow101>
set it to true
< cobega>
using ./bitcoin-cli importmulti '[{ "address": "2MtxopLjtoJbfxJqNptZR17RzPGSTSTmqZsy", "timestamp": 1546218000 }]' I get this error Invalid scriptPubKey, as far as I see the docs, it's either scriptPubKey or address to use?
< achow101>
no, it's {"scriptPubKey":{"address":"..."}}
< cobega>
using it like ./bitcoin-cli importmulti '[{"scriptPubKey":{"address":"2MtxopLjtoJbfxJqNptZR17RzPGSTSTmqZsy"},"timestamp":1546218000}]' it says "invalid address"
< cobega>
Using testnet while development of application.
< arubi>
the 2Mtxop one is invalid
< cobega>
shame on me. works as expected :)
< arubi>
:)
< gwillen>
sipa: except that addmultisigaddress only sort of imports
< gwillen>
it makes it solveable but it doesn't make it "watchonly", i.e. look for utxos it owns
< sipa>
gwillen: yes, you need importaddress for that
< sipa>
addmuktisigaddress predates support for watchonly addresses
< sipa>
it uzed to only work in case you had *all* the private keys for a multisig address in your wallet, at all
< sipa>
(yes, that makes no sense)
< cobega>
Anybody a bit more familiar with curl? Using CLI my command works fine, but for curl I receive: curl --user user:pass --data-binary '{"jsonrpc": "1.0", "id":"", "method": "importmulti", "params": [{"scriptPubKey":{"address":"2MtxopLjtoJbfxJqNptZR17RzPGSTSTmqZy"},"timestamp":1546218000},{"rescan":true}] }' -H 'content-type: text/plain;' http://domain.com:28339/ {"code":-3,"message":"Expected type array, got object"}
< fabianfabian>
I think your first param is an object but should be an array