<sipa>
which on its turn invokes the normal autogen/configure/make build scripts
<sipa>
(which again, you can invoke yourself, if you just want to build for development, and that's also what i'd recommend)
<fanquake>
note that depends itself is also deterministic, just that if you use it locally it's not deterministic against other builders
<sipa>
^
<Davidbak>
just curious - are there any discussions anywhere on why the build system is the way it is?
<fanquake>
due to differing toolchains etc
jarthur has joined #bitcoin-core-dev
<sipa>
guix is _just_ replacing an older system called gitian
<sipa>
which pretty much was the first thing to even try reproducible building anywhere
jarthur_ has quit [Ping timeout: 260 seconds]
<sipa>
for bitcoin core we really want reproducibility, because of the view that maintainers should be trusted the least amount possible
<sipa>
e.g. no auto-update (because whoever has the keys to push an update could hijact the network's consensus rules, pretty much)
<sipa>
the reproducibility helps, because it means we can commit to specific published binaries, and anyone can attest to the fact that these binaries indeed correspond to the source code available on github (and mirrors)
<sipa>
in fact, the release process involves various contributors publishing such GPG attestations
<Davidbak>
well reproducible builds are good for many reasons, bitcoin obviously depends on that for trust (and reliability of the consensus) but over the years there _have_ been other tools developed, that have gained traction in the overall C++ community, that address that well. As a newbie I AM NOT COMING IN AND CRITICIZING or saying "Why aren't you doing thing in this obvious better way, you old fogies!?" - but as a very long
<Davidbak>
time C++ dev with lots of commercial experience yet still not UNIVERSAL development experience I'm always look to learn the pros/cons of different programming styles, software engineering techniques, build systems, etc etc.
bitdex has quit [Ping timeout: 276 seconds]
<fanquake>
reproducibility is also good because the idea that you could end up with two differently functioning monies from the same source code is just nuts
<sipa>
davidbak: as far as i know, actual reproducible builds in C++ is pretty recent
bitdex has joined #bitcoin-core-dev
<sipa>
until a few years ago, GCC used randomness at compile time; now it exposes a way to set a specific seed for that randomness
<gene>
was the randomness used for name mangling?
<gene>
s/was/is
<sipa>
bitcoin core contributors have contributed patches to various building/system tools to make sure nothing breaks the reproducibility
<Davidbak>
oh yeah, because just to name one thing, compilers/linkers love to insert timestamps into object files and binary executables, but the _goal_ has been around since embedded systems started using C a long long time ago and people addressed it in various ways: locking down toolchains, locking down external libraries, etc. etc. all trying to reach accountability - even if bit-for-bit binary reproducibility wasn't achievable.
<Davidbak>
i do get all this - and plenty of people are working on reproducible builds, not just bitcoin - and it still isn't available out-of-the-box without work
<Davidbak>
but, _because_ there are other people working on it, there are multiple solutions.
<sipa>
i don't really know of any off the shelf usable ones for us
<sipa>
i mean, besides just running the build in a fixed container - which is what we used to do with gitian
<sipa>
but then you get the question of where the container comes from etc
<Davidbak>
well as far as _build systems_ go: bazel, to name one, can give you reproducible builds given that your toolchains can give you reproducible builds. it's functional, like Nix, and requires all dependencies to be explicit and locked down, and it really works.
<fanquake>
There may be multiple solutions, but Bitcoin Core has had a solution to this problem for ~10 years now, and Guix is a good next step.
<sipa>
davidbak: oh, that's not the issue
<Davidbak>
but i'm not proselytizing, i'm just curious/interested.
<Davidbak>
what's not the issue?
<sipa>
davidbak: that won't give you reproducibility between different systems that have different compiler versions
Kaizen_Kintsugi has joined #bitcoin-core-dev
<Davidbak>
how does the bitcoin build system give you that?
<Davidbak>
an arm build is obviously different than an x64 build, a windows build different from a ubuntu build ...
<sipa>
davidbak: guix gives us that, because it compiles the compiler, bootstrapped from the smallest possible piece of binary that exists (to the best of my knowledge)
<sipa>
at an exactly specified version
<sipa>
along with all system libraries etc
<fanquake>
and then all of our dependencies are *locked down* in our depends system
<gene>
isn't Bazel a facebook product?
<sipa>
i think bazel would be a reasonable alternative for our hand-rolled depends system
<sipa>
except that it'd imply (yet) another build system to maintain, as afaik it doesn't integrate with autotools/configure/makefiles
<Davidbak>
bazel is the open-source version of a google internal product - though well supported (and most of the contributors are googlers afaik) facebook has a _different_ c++ build system, called buck. As I said, there's a _lot_ of work in this space
<sipa>
in any case, this isn't the hard part
<Davidbak>
no, bazel is a world of its own, doesn't like to operate with any other build system at all. It's all or nothing, pretty much!
<Davidbak>
so, as I said, I'm not shilling for it. Just wanted to know the tradeoffs.
<sipa>
the hard part is an exactly reproducible build environment
<Davidbak>
And if guix builds from the compiler - or even more importantly, the libc/libc++ layer UP, that's pretty damn locked down, and i don't think that's a typical use case for bazel (possibly not even supported, don't know)
<Davidbak>
(i mean, _including_+ the libc/libc++ layer)
<sipa>
davidbak: well, yes, i'm not lying when i say you and i, on our home desktop systems, on different linux distributions, with different versions of everything, will very realistically succeed in producing bit for bit identical bitcoin core binaries if you run the guix-based build system
<Davidbak>
i'm definitely looking forward to seeing that happen!
Kaizen_Kintsugi has quit [Ping timeout: 260 seconds]
<sipa>
well, it's how it works today
<sipa>
22.0 was built and released this way
<Davidbak>
i get you that it works today! and when I get it running here on my system, I'll have seen it happen!
<sipa>
ah, yes, i encourage you try
<fanquake>
apt install guix && ./contrib/guix/guix-build and you should be good to go
<sipa>
oh, and about the different systems: everything is cross-compiled from the Linux environment guix offers (including windows binaries using mingw, and macOS binaries using apple-patched clang)
<Davidbak>
amazing! well, off I go to build a VM for it (as a first cut, not ready to try it in WSL until it's working "native") and that's interesting about the windows binaries too - BUT!!!
<sipa>
ha, i wonder if guix would work in WSL... not unreasonable that it does, i think
<sipa>
(but i have no clue)
<Davidbak>
but w.r.t. the windows build - there are instructions for building with Visual Studio/MSVC. Which is obviously NOT mingw. So if you build that way (for development purposes) that's not going to be binary compatible. So that must be for developers only, not for release?
<sipa>
exactly
<Davidbak>
ok
<sipa>
it's not too different from just being able to build with make, using whatever compiler you want (e.g. clang on linux, while the release binaries are built with gcc)
<sipa>
plus, having CI test more build environments than just those used for release builds helps find certain issues w.r.t. language violations in the codebase or unintentional architecture dependencies
<sipa>
at some point we had a CI target that involved building for, and testing in a CPU emulator, an s390x target, because it's big endian
<sipa>
but iirc it was too slow
<Davidbak>
Yes. Different compilers == different warnings/errors, especially as the C++ standard allows a bunch of stuff to go undiagnosed and be "implementation defined" but building with multiple compilers increases your chance that at least one of them will warn you of a problem like that.
<Davidbak>
Huh. So the mainframe guys wanting to run bitcoin core on big iron might run into incompatibilities, eh?
<Davidbak>
Can't say that's a use case I personally would spend much time on ....
<sipa>
no, neither am i
<sipa>
but i am interested in knowing that our codebase doesn't unintentionally assume little-endianness anywhere, because if it does, that's a red flag for undefined/unspecified behavior
<Davidbak>
ok, i'm off - I appreciate your time (and of course, your information)!
<sipa>
yw
<Davidbak>
yes - but ... i'm sure there's a static analysis for that _somewhere_?
<sipa>
yes, presumably, but afaik most of those tools are proprietary
<Davidbak>
one more comment: i especially find the endianness issue funny for bitcoin because in addition to the usual on-the-wire format vs. internal representation issues you typically get bitcoin also has - for some unknown historical reason? - the issue of people _visualizing_ addresses/hashes/etc differently than the internal representation - how the heck did _that_ happen?
<sipa>
building/testing with sanitizers also catches most of such issues
<sipa>
but who knows - maybe not everything
<sipa>
davidbak: it's actually perfectly consistent if you know the history ;)
<sipa>
bitcoin historically treated all byte arrays as little-endian numbers
<sipa>
it also treated SHA256 as a black box - something you send bytes, and it gives 32 bytes - which are then interpreted as a little-endian number (just like anything else)
<Davidbak>
(history: looking forward to the "retrospective" papers/books being published about 10yrs from now ...)
<Davidbak>
ok, so far you're all little-endian THE WAY IT SHOULD BE! - what's next in the history?
<sipa>
the same is true for other cryptographic constructs (e.g. ECDSA signatures contain big-endian serialized numbers, but from bitcoin's perspective, they're just byte arrays)
Kaizen_Kintsugi has joined #bitcoin-core-dev
<sipa>
now, in block hashes, the hash is actually interpreted as a number, in order to compare it with the target (for proof of work)
<sipa>
so the _number_ was printed for human consumption - who typically want largest digit first
<sipa>
if you forget that it's a number, a think it's a hexdump of a 32-byte array, that means it indeed looks swapped compared to the wire format
<sipa>
but if you think of it as a hex number for human consumption, it's exactly right
<sipa>
... of course, that method of formatting hashes was then used for everything, including txids and many other things, where the numerical representation is meaningless
<Davidbak>
so the confusion is that the consumers (I guess) of hashes for the purpose of looking at "how many leading zeros" they have (already an approximation to the truth) aren't PROGRAMMERS!
<sipa>
i like your thinking
<Davidbak>
sadly, it's the only thinking I have ...
<sipa>
there is AFAIK one exception to "everything is little endian"... the TCP port rumoured in addr messages on the network is big endian
<Davidbak>
I've been thinking this way for 50 years, ever since my 9th grade math teacher - a laid-off rocket engineer (seriously) - got bored teaching math one day and gave us a 45 minute class in FORTRAN.
<sipa>
because network stuff is always big endian for some reason, and i think the original code just copied the bytes
<Davidbak>
network is always bigendian for historical reasons
<sipa>
yeah, i'm aware
<sipa>
ntohs and friends make that clear
Kaizen_Kintsugi has quit [Ping timeout: 252 seconds]
<Davidbak>
anyway, gotta go - speak to you again after I get this set up ... or ... if I run into trouble!
<sipa>
cya
<gene>
./
c_arc has joined #bitcoin-core-dev
jarthur_ has joined #bitcoin-core-dev
Kaizen_Kintsugi has joined #bitcoin-core-dev
jarthur has quit [Ping timeout: 252 seconds]
Kaizen_Kintsugi has quit [Ping timeout: 245 seconds]
hsmiths has quit [Quit: Connection closed for inactivity]
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
earnestly has quit [Ping timeout: 252 seconds]
ghostkeeper has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
ghostkeeper has quit [Ping timeout: 252 seconds]
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
gene has quit [Quit: gene]
grettke has quit [Ping timeout: 252 seconds]
c_arc has joined #bitcoin-core-dev
grettke has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
ghostkeeper has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
Kaizen_Kintsugi has quit [Remote host closed the connection]
Kaizen_Kintsugi has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
Kaizen_Kintsugi has quit [Ping timeout: 252 seconds]
c_arc has joined #bitcoin-core-dev
ghostkeeper has quit []
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
ghost43 has quit [Quit: Leaving]
ghost43 has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc_ has joined #bitcoin-core-dev
c_arc has quit [Remote host closed the connection]
<bitcoin-git>
bitcoin/master fa53d3d MarcoFalke: test: Check that bitcoin-tx accepts whitespace around sequence id and mult...
<bitcoin-git>
bitcoin/master fafab8e MarcoFalke: bitcoin-tx: Reject non-integral and out of range sequence ids
<bitcoin-git>
bitcoin/master fa6f29d MarcoFalke: bitcoin-tx: Reject non-integral and out of range multisig numbers
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
bitcoin-git has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] laanwj merged pull request #23253: bitcoin-tx: Reject non-integral and out of range int strings (master...2110-utilTxSeqId) https://github.com/bitcoin/bitcoin/pull/23253
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
ArctVaulMarsHMPJ has quit [Quit: EXIT]
jesseposner_ has quit [Ping timeout: 250 seconds]
jesseposner has joined #bitcoin-core-dev
jrayhawk has quit [Ping timeout: 250 seconds]
jrayhawk has joined #bitcoin-core-dev
Kaizen_Kintsugi has joined #bitcoin-core-dev
paulo has quit [Ping timeout: 240 seconds]
TheRec has quit [Ping timeout: 245 seconds]
TheRec has joined #bitcoin-core-dev
paulo has joined #bitcoin-core-dev
goatpig has quit [Remote host closed the connection]
sturles_ has quit [Ping timeout: 245 seconds]
sturles has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
adiabat has quit [Ping timeout: 245 seconds]
c_arc has joined #bitcoin-core-dev
adiabat has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
theStack has quit [Ping timeout: 252 seconds]
theStack has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
ArctVaulMarsHMPJ has joined #bitcoin-core-dev
adiabat has quit [Ping timeout: 265 seconds]
<jnewbery>
sipa: did you see cfields question about a secp c++ wrapper earlier? I'm also curious!
<jnewbery>
< cfields> sipa: what do you think about a c++ shim-wrapper (like aes.cpp for ctaes) around secp for Core, so we could build it as c++? With tighter integration, that'd save us from having to add all of the c buildsystem machinery for the single secp object.
<sipa>
i did see it, but forgot to respond
<sipa>
maybe a discussion to have in #secp256k1 ?
cfields has quit [Remote host closed the connection]
Kaizen_Kintsugi has quit [Ping timeout: 252 seconds]
cfields has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
adiabat has joined #bitcoin-core-dev
ziggie has quit [Quit: Connection closed for inactivity]
c_arc has joined #bitcoin-core-dev
Keele has quit [Ping timeout: 265 seconds]
c_arc has joined #bitcoin-core-dev
<laanwj>
that would be a nice buildsystem simplification
c_arc has joined #bitcoin-core-dev
hsmiths has joined #bitcoin-core-dev
Guyver2 has joined #bitcoin-core-dev
Davidbak has quit [Quit: Leaving]
bairen has quit [Ping timeout: 276 seconds]
bairen has joined #bitcoin-core-dev
Kaizen_Kintsugi has joined #bitcoin-core-dev
Guest8 has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
Guest8 has quit [Quit: Client closed]
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
hex17or has quit [Remote host closed the connection]
hex17or has joined #bitcoin-core-dev
bitcoin-git has joined #bitcoin-core-dev
<bitcoin-git>
[gui] laanwj opened pull request #454: qt: Use only Qt translation primitives in GUI code (master...2021-10-qt-use-at-translation) https://github.com/bitcoin-core/gui/pull/454
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
c_arc has joined #bitcoin-core-dev
bairen has quit [Remote host closed the connection]
<bitcoin-git>
[bitcoin] stratospher opened pull request #23271: crypto: Fix K1/K2 use in the comments in ChaCha20-Poly1305 AEAD (master...fix-k1-k2) https://github.com/bitcoin/bitcoin/pull/23271
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
c_arc has joined #bitcoin-core-dev
pigeons_ is now known as pigeons
jamesob has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
<cfields>
fwiw, the conclusion from #secp256k1 was that compiling as c++ would be too risky, given that it was written and will continue to be written against exactly the c89 language.
dodo__ has joined #bitcoin-core-dev
dodo has quit [Ping timeout: 245 seconds]
yanmaani has quit [Ping timeout: 276 seconds]
promag has joined #bitcoin-core-dev
promag__ has quit [Read error: Connection reset by peer]
c_arc has joined #bitcoin-core-dev
gnaf has quit [Quit: Konversation terminated!]
Keele has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
gnaf has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
AaronvanW has quit [Quit: Leaving...]
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
Kaizen_Kintsugi has quit [Remote host closed the connection]
Kaizen_Kintsugi has joined #bitcoin-core-dev
meshcollider has quit [Remote host closed the connection]
paulo has quit [Remote host closed the connection]
Kaizen_Kintsugi has quit [Ping timeout: 245 seconds]
Kaizen_Kintsugi has joined #bitcoin-core-dev
Talkless has quit [Quit: Konversation terminated!]
Kaizen_Kintsugi has quit [Ping timeout: 252 seconds]
c_arc has joined #bitcoin-core-dev
meshcollider has joined #bitcoin-core-dev
paulo has joined #bitcoin-core-dev
Kaizen_Kintsugi has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
dodo__ is now known as dodo
___nick___ has quit [Ping timeout: 252 seconds]
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
yanmaani has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
bitcoin-git has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] hebasto closed pull request #22764: build: Include qt sources for parsing with extract_strings.py (master...210821-translation) https://github.com/bitcoin/bitcoin/pull/22764
bitcoin-git has left #bitcoin-core-dev [#bitcoin-core-dev]
Kaizen_Kintsugi has quit [Remote host closed the connection]
Kaizen_Kintsugi has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
Kaizen_Kintsugi has quit [Ping timeout: 252 seconds]
c_arc has joined #bitcoin-core-dev
jarthur has joined #bitcoin-core-dev
Kaizen_Kintsugi has joined #bitcoin-core-dev
<shiza>
What's the risk in feeding C89 to C++ compilers?
<sipa>
the languages are different
<sipa>
valid C89 Cc may exhibit ubdefined or implementation defined behavior in C++
<sipa>
that was garbled
<shiza>
There must be two or three things in C89 that not every C++ compiler is capable of translating.
<sipa>
being able to translate is not the issue
<shiza>
Can't have CI warning them?
<sipa>
not if we don't know about them
Guyver2 has quit [Quit: Going offline, see ya! (www.adiirc.com)]
<shiza>
Do you mean future C++ standards?
<shiza>
Can C++ drop C89 elements while keeping FFI essences?
<sipa>
the issue isn't language features, but rules
<sipa>
C++ for example is stricter about aliasing and alignment than C is
<sipa>
valid C89 code may be simultaneously compilable by actual C++ compilers but still have undefined behavior
<sipa>
and i'm sure an expert in both languages could give an exhaustive list of such differences, and we could go over the source code to make sure it isn't the case
<sipa>
but even then, it places a burden on future libsecp contributors to be simultaneously satisfying all C++'s rules; and those comtributors are mostly C programmers
* shiza
nods in a mix of ignorance and agreement
c_arc has joined #bitcoin-core-dev
<shiza>
Ugh now I hit the large #secp conversation about it.
AaronvanW has joined #bitcoin-core-dev
ghostkeeper has quit []
c_arc has joined #bitcoin-core-dev
meshcollider has quit [Changing host]
meshcollider has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
<theStack>
TIL that it is currently not possible to send a serialized message (CSerializedNetMsg instance) to multiple peers
<theStack>
CConnman::PushMessage uses move semantics... so after one call it's like "aaaaand it's gone" ;)
<theStack>
not a big deal, but that seems to be the reason the TODO in PeerManagerImpl::NewPoWValidBlock is still there "TODO: Avoid the repeated-serialization here"
Kaizen_Kintsugi has quit [Remote host closed the connection]
Kaizen_Kintsugi has joined #bitcoin-core-dev
Kaizen_Kintsugi has quit [Remote host closed the connection]
Kaizen_Kintsugi has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
Kaizen_Kintsugi has quit [Ping timeout: 252 seconds]
ArctVaulMarsHMPJ has quit [Quit: EXIT]
c_arc has joined #bitcoin-core-dev
rex4539 has joined #bitcoin-core-dev
morcos_ has joined #bitcoin-core-dev
morcos has quit [Ping timeout: 276 seconds]
morcos_ is now known as morcos
rex4539 has quit []
Kaizen_Kintsugi has joined #bitcoin-core-dev
Kaizen_Kintsugi has quit [Ping timeout: 252 seconds]
bitdex has joined #bitcoin-core-dev
gleb7 has quit [Quit: Ping timeout (120 seconds)]
gleb7 has joined #bitcoin-core-dev
Kaizen_Kintsugi has joined #bitcoin-core-dev
ArctVaulMarsHMPJ has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
Kaizen_Kintsugi has quit [Ping timeout: 252 seconds]
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
Kaizen_Kintsugi has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
Kaizen_Kintsugi has quit [Ping timeout: 252 seconds]
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
roconnor has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
c_arc has joined #bitcoin-core-dev
belcher has joined #bitcoin-core-dev
gleb7 has quit [Read error: Connection reset by peer]