< miketwenty1>
aww i see.. "it must be signed by a recognized certificate authority. Anyone who distributes malware under a valid certificate is held legally accountable for the software they distribute."
< miketwenty1>
makes sense
< miketwenty1>
so you would sign the binaries and places like bitcoin-core.org would have access to achow101's signed exe.. if i follow
< sipa>
all the release binaries are signed
< sipa>
but they're created in a 3-step gitian process; (a) people build and agree on an unsigned build (b) achow signs the binary that comes out and publishes the signature (c) people create a signed build by combining their unsigned binaries with the posted signature
< miketwenty1>
i look forward to learning the process of (c)
< luke-jr>
[00:01:06] <miketwenty1> aww i see.. "it must be signed by a recognized certificate authority. Anyone who distributes malware under a valid certificate is held legally accountable for the software they distribute."
< achow101>
you should have a read through of that
< miketwenty1>
achow101: thanks for the link i looked at this earlier but missed the section "After 3 or more people have gitian-built and their results match: " .. now that 3 PR's have been pulled in.. and mine has been sitting as the 4th.. i should probably do this bit?
< achow101>
that's for the release maintainer to do
< achow101>
Currently we are basically at the "Next steps" header
< achow101>
right where it says "codesigner only"
< miketwenty1>
makes sense
< sipa>
specifically, you are now at the "Non-codesigners: wait for Windows/macOS detached signatures:" stage
< achow101>
the builders stop at the end of that section, after pushing their sigs for the gitian.sigs repo
< * luke-jr>
wonders if we should be trying to add support for Apple ARM systems
< fanquake>
There should be a few updates on the macOS Guix front shortly
< luke-jr>
wumpus: don't see anything there related?
< luke-jr>
wumpus: I mean macOS ARM
< luke-jr>
that they just started selling recently
< wumpus>
no idea then
< wumpus>
macos ARM used to be iOS
< wumpus>
I don't exactly keep day to day track of what apple is doing fwiw :-)
< sipa>
are they actually for sale already?
< sipa>
i thought they were just announced
< wumpus>
but of course, if that becomes their new architecture then we should support that too
< fanquake>
according to the apple website I can buy one now
< achow101>
they're on sale now
< sipa>
i just mean it's probably hard to for us test anything as long as none of us have hardware
< sipa>
unless there is some sort of simulator
< achow101>
I thought provoostenator got the dev kit
< sipa>
ah cool
< achow101>
in theory we shouldn't need to do anything for now because of their x86 emulator
< luke-jr>
achow101: good luck with IBD
< luke-jr>
x86->ARM isn't going to be like m68k->PPC or PPC->x86
< luke-jr>
even if they got comparable performance, emulation is going to have a very noticable impact, especially on CPU-intensive work
< wumpus>
native build is going to be easy, I expect, building for ARM is also not a challenge, the difficult part is likely what new things need to be reverse engineered to cross-compile for it from gitian
< wumpus>
e.g. for their SDKs and strange toolkits
< luke-jr>
wumpus: I kinda wonder if we can just change the CHOST target…
< luke-jr>
I guess probably need to update the Xcode too
< wumpus>
new kinds of 'fat executables' maybe !
< luke-jr>
ugh
< fanquake>
"Universal binaries"
< luke-jr>
true, I guess we probably can't just ship two different DMGs? XD
< achow101>
they might be enforcing notarization for arm things too?
< luke-jr>
that'd suck
< luke-jr>
I guess in that case we have no choice but to go along with it
< luke-jr>
firstrun dialog warning users of Apple'
< luke-jr>
's privacy violations?
< wumpus>
they really should have skipped ARM and went straight to RISC-V, of course this means they'll have to go through the process again in a few years :-)
< luke-jr>
why would Apple care to go RISC-V?
< wumpus>
I wasn't entirely serious
< wumpus>
buut just there seems to be some kind of pattern there
< sipa>
the licensing costs and closedness of ARM isn't an issue for them :)
< achow101>
they've been making arm chips for years now
< sipa>
also that
< luke-jr>
I wonder if they're dropping x86 simply to consolidate back to 1 arch
< achow101>
dunno about consolidate, but it's definitely to make the walled garden even more walled
< * luke-jr>
ponders if there's any decompilers that take advantage of multiple archs
< achow101>
wdym by "take advantage"?
< wumpus>
yes it makes sense from an economic point of view, ARM chips are produced at much larger quantities, it's probably just a lot cheaper for them
< luke-jr>
achow101: presumably it could infer more about the code
< luke-jr>
maybe not enough to be worth the additional effort tho
< wumpus>
why would you want to decompile apple's garbage anyway
< * luke-jr>
shrugs
< luke-jr>
it's the kind of thing you'd only do for fun
< wumpus>
heh
< achow101>
luke-jr: there's decompilers for a lot of archs
< luke-jr>
achow101: not quite the same thing
< achow101>
not sure what you mean then
< luke-jr>
achow101: I mean looking at both the x86 and ARM builds of the same thing, to give better decompilation output
< achow101>
ah. the decompilers I know of only take one binary at a time so they can't do that
< luke-jr>
right
< wumpus>
I do understand luke-jr's point, if you have the guarantee that the same source code is compiled to different architectures and binary formats, that might help learning more about it because the compiler might do slightly different (but deterministic) things
< wumpus>
that said, the point of decompilation is rarely to reconstruct as much as possible about the original code, just to have a easier to read representation than assembly language
< sipa>
if decompilers could do that, i'm sure they'd try
< sipa>
actually, maybe not, if it were possible to reconstruct something that was more readable than the original code, but not identical, they'd do that
< wumpus>
I read something funny about Turbo C 2.0 (or similar) once, that compiler generated such predictable code that it's possible to reconstruct the original C code (besides variable names and comments ofc)
< wumpus>
sipa: exactly :)
< luke-jr>
[02:57:55] <luke-jr> it's the kind of thing you'd only do for fun
< sipa>
in such i world i expect proprietary software to be littered with dummy inline assembly
< wumpus>
there is the business case 'I lost the source code and want to reconstruct it' but I expect it's a fairly rare use of decompilers, most use is for figuring out malware
< sipa>
wumpus: do you know openttd?
< wumpus>
sipa: yes!
< luke-jr>
wumpus: meh, don't really even need a decompiler for that
< wumpus>
didn't know it was based on decompilation though
< luke-jr>
last time I cracked DRM, it was trivial to work from objdump
< achow101>
only time i've used a decompiler is for a ctf
< wumpus>
luke-jr: no on 'needs' a decompiler, it's kind of a luxury but can definitely speed up things (for people that have to do that work all day)
< luke-jr>
capture the flag?
< achow101>
yes
< luke-jr>
wumpus: well, it didn't take even an hour
< luke-jr>
achow101: how do you ctf with a decompiler? O.o
< wumpus>
sure, of course...
< miketwenty1>
achow101: have you ever done the ctf's at defcon?
< luke-jr>
I basically changed a memcmp call to printf or something
< wumpus>
'I can do it all from the raw hex, blindfolded!'
< miketwenty1>
i ask because i noticed emzy isn't on there
< wumpus>
miketwenty1: the idea is that people add themselves
< miketwenty1>
k
< fanquake>
after they’ve done a number of builds
< miketwenty1>
is there a magic number before i should add myself fanquake?
< wumpus>
if emzy is building but not on there you could prod them to add themselves
< wumpus>
3 or so
< achow101>
emzy is there, but maybe that's a different key?
< miketwenty1>
ohh maybe emzy is there.. but i was looking at the names..
< luke-jr>
XD
< miketwenty1>
nvm
< fanquake>
Yea 2 or 3. I’m pretty sure it’s in the docs
< miketwenty1>
Stephan Oeste (Emzy)
< sipa>
ah, so the Apple ARM stuff needs macOS 11, not 10.x
< miketwenty1>
i just got done watching mandalorian.. im thinking if i get my name on this list i can get my beskar armor
< sipa>
so i guess we'll need to build that with a later xcode
< sipa>
but apart from that, i do expect it to be simple (as long as Qt etc support it)
< wumpus>
right it doesn't have to be a great number of builds just to prove that a) you can do gitian builds b) you're willing to do them regularly, just to make sure you're not wasting people's time adding the signature
< achow101>
i expect apple will have done something that just makes it super painful
< sipa>
achow101: not unlikely :)
< wumpus>
we can't do the cross build with xcode, the difficulty is going to be the open source cross toolchain
< sipa>
oh right, there may be complications from running an m1/osx-compatible clang on linux
< sipa>
osxi i guess...
< wumpus>
which needs to produce whatever kind of new binary format they've come up with and maybe they have some fun linker extensions too and whoknows
< wumpus>
yeah
< wumpus>
even the plists are still a continous source of fun
< sipa>
well, all things we can figure out once someone has hardware
< wumpus>
I think that's a good way to go about it, no need to be overly eager anyhow, wait a bit for the initial bugs and tooling issues for the platform to be ironed out by others
< fanquake>
Yea. Like having your macbook bricked after upgrading to macOS 11
< wumpus>
sorry for your loss :/
< fanquake>
heh no, I'm remaining firmly on macOS 10. I'll think about an upgrade after a few point releases.
< fanquake>
Just seen reports of bricking in older macbooks
< wumpus>
I really prefer not to see it on the last minute PRs list for the 22.0 milestone *again* :-)
< wumpus>
I'm sure we can simplify things like that later
< miketwenty1>
has anyone here used terraform?
< wumpus>
fanquake: but agree it's become a horrible mess; at some point we'd likely want to do real ELF parsing instead of parsing the output of readelf, I think in many cases it's easier
< wumpus>
no need for regexps in any case
< luke-jr>
regexps are better than error-prone parsing :x
< wumpus>
they're just as error prone just hide this by being inscrutinable too
< wumpus>
in any case, parsing command output text was the wrong way to go about this (this is my fault, it seemed easier when it was still a few simple checks, it always starts out like that doesn't it)
< luke-jr>
parsing ELF sounds a lot harder tho
< wumpus>
it's really easy
< wumpus>
just some fixed binary headers
< wumpus>
that we only handle a small subset of possible architectures makes it even simpler
< sipa>
yeah and then some architecture is added with 9-bit bytes, and all your parsing code breaks!
< wumpus>
I'm sure readelf would choke on that too :)
< sipa>
wumpus: are you up early, or up late?
< wumpus>
but joking aside the ELF format has been really stable since, 1995 or so (when spec 1.2 was released), of course there have been some platform-specific extensions but the essence is still exactly the same, there's not many data formats that stable
< wumpus>
sipa: very early
< fanquake>
wumpus: it did feel a bit too early for you to be merging things
< sipa>
wumpus: excited for 0.21 huh :p
< wumpus>
sipa: kind of! mostly excited for the feature merge window to be open again, but yes :p
< sipa>
ok, excited for 0.22!
< fanquake>
already into all the cool new stuff for 22.0
< wumpus>
right
< wumpus>
i just rolled out of bed and had some energy and motivation which is great, will probably crash later
< wumpus>
most of the security checks are literally just 'check a bit flag in some header', some check for presence of a symbol, I guess reading strings (say, for section names) is the only variable length operation but also a very straightforward one
< fanquake>
who wants to upload a dmg somewhere
< achow101>
fanquake: already doing that, just slowly
< wumpus>
I don't have macosx output yet
< luke-jr>
wumpus: looks like it's symobl versions
< luke-jr>
wumpus: it's parsing: 4: 0000000000000000 0 FUNC GLOBAL DEFAULT [<localentry>: 8] UND memcpy@GLIBC_2.17 (2)
< wumpus>
well-if you load the symbol table directly at least all those fields will be at fixed offsets within a record, no ambiguity what starts where, the symbol name location is always at the same offset
< luke-jr>
wumpus: true
< wumpus>
this could be the case for text output too, but readelf definitely doesn't heed that, column sizes change all the time; columns differ per architecture; it's clearly for human consumption not machine parsing
< achow101>
so the fix is to apply that commit ourselves during depends build?
< fanquake>
The issue is that we download a prebuilt Clang. So we'd have to compile ourselves if we want that fix. I don't think there is an 8.x version available that has it
< achow101>
is 8.x required?
< fanquake>
It was picked to "line up" with other macOS tools and the SDK
< fanquake>
I've pinged Carl/Cory, they might already have something in mind, but will be sleeping
< achow101>
do you know what exactly the nature of the nondeterminism is? istm we should all have different osx results
< fanquake>
Not exactly sorry. From the discussion I saw they were digging somewhere in qpaintengine
< achow101>
fanquake: can you link me to the discussion?
< fanquake>
achow101: I would but it's in a group chat, & not public
< fanquake>
It's just where we occasionally talk about "cool" & exciting build system stuff heh
< achow101>
these build people doing things behind closed doors :p
< achow101>
I thought that's what #bitcoin-builds was for
< fanquake>
Yea we probably need to start diverting some traffic into that
< Kiminuo>
MarcoFalke, Hi, would you possibly know how to add `-lstdc++fs`. It seems I need that to avoid linking errors on Ubuntu 18.04. I compile like this: ./autogen.sh && ./configure CC=gcc-8 CXX=g++-8 BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include" && make -j10
< bitcoin-git>
[bitcoin] jonasschnelli opened pull request #20430: sanitizers: Add suppression for unsigned-integer-overflow in libstdc++ (master...2020/11/suppressions) https://github.com/bitcoin/bitcoin/pull/20430
< jonasschnelli>
Oh. Our apple developer program expires in 9 days. Better renew now.
< hebasto>
catching up the macos non-deterministic gitian builds (4 sigs vs 2 sigs). Do I understand correctly that we blame llvm?
< fanquake>
hebasto: that's correct. I'll make sure to summarize in an issue on the repo once I've talked to Carl / Cory. They were debugging the same issue earlier from what I can tell.
< hebasto>
fanquake: thanks
< jonasschnelli>
fanquake: is 0.21.0rc1 mac non deterministic? Shall I wait with the code signatures?
< fanquake>
jonasschnelli: yea it is. We think it's an issue in LLVM 8. I don't think you necessarily have to wait with the sigs, as the fix could be involved, and we've had non-determinism in an rc before. However up to you.
< fanquake>
We've already got an rc2 happening because of some wallet fixes, and now the non-determinism, so we could probably even just skip the signed binaries for rc1
< jonasschnelli>
ack.. thanks. I just read the backlog.
< jonasschnelli>
I'll skip rc1
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #20431: [backport 0.21] tests: shrink feature_taproot transfer of funds tx (0.21...202011-test-taproot-signmany) https://github.com/bitcoin/bitcoin/pull/20431
< bitcoin-git>
bitcoin/0.21 7ffac12 Anthony Towns: tests: shrink feature_taproot transfer of funds tx
< bitcoin-git>
bitcoin/0.21 bf9548b MarcoFalke: Merge #20431: [backport 0.21] tests: shrink feature_taproot transfer of fu...
< bitcoin-git>
[bitcoin] MarcoFalke merged pull request #20431: [backport 0.21] tests: shrink feature_taproot transfer of funds tx (0.21...202011-test-taproot-signmany) https://github.com/bitcoin/bitcoin/pull/20431
< MarcoFalke>
Glad we pushed the rc1 yesterday. Was sort of expected to hit a gitian issue
< provoostenator>
I just pushed mine. macOS is the same as luke-jr and fanquake
< provoostenator>
I think my gitian tooling is up to date, since I tested PR's related to macOS (SDK) changes
< bitcoin-git>
[bitcoin] laanwj opened pull request #20434: contrib: Parse ELF directly for symbol and security checks (master...2020_11_pixie) https://github.com/bitcoin/bitcoin/pull/20434
< fanquake>
heh pixie
< Kiminuo>
hmm, this seems to be the magic formular to compile code <filesystem> on Ubuntu 18.04: "g++-8 --std=c++17 main.cpp -lstdc++fs" ... now there must be an easy way to specify it for Bitcoin core build. I have tried: ./autogen.sh && ./configure CC=gcc-8 CXX="g++-8" LDFLAGS="-lstdc++fs" --with-incompatible-bdb && make -j10
< Kiminuo>
but apparently that's not it
< Kiminuo>
any tips?
< wumpus>
😈
< fanquake>
it took me a second
< Kiminuo>
Hm, not funny :D
< wumpus>
Kiminuo: weird, LDFLAGS should work to specify an additional library
< wumpus>
Kiminuo: you've checked make V=1 to see if it doesn't get passed at all?
< Kiminuo>
"g++-8 --std=c++17 -lstdc++fs main.cpp" this fails with the similar errors as " ./autogen.sh && ./configure CC=gcc-8 CXX="g++-8" LDFLAGS="-lstdc++fs" --with-incompatible-bdb && make -j10"
< fanquake>
Cory has confirmed that the macOS determinism issue is the same one him and Carl had seen, and bisected to a commit in LLVM. Possible that we might be able to get away with a workaround in Qt rather than having to either switch to a newer compiler, or build our own.
< fanquake>
He said he could open an issue on the repo with some more details.
< wumpus>
Kiminuo: if it also doesn't work for a stand-alone main.cpp, then maybe the workaround doesn't work
< Kiminuo>
g++-8 --std=c++17 main.cpp -lstdc++fs
< Kiminuo>
this works
< Kiminuo>
wumpus, or I should say, it does not report any errors
< wumpus>
so it needs to be the last argument? okay, I don't think you're going to be able to do that without makefile changes
< bitcoin-git>
[bitcoin] MarcoFalke merged pull request #20223: build: Drop the leading 0 from the version number (master...drop-leading-0) https://github.com/bitcoin/bitcoin/pull/20223
< MarcoFalke>
vasild: One does not simply open a pr without opening a can of worms
< vasild>
some of the compile errors on CI are "#include <filesystem> - file not found", so will not be fixed by linking with an extra library stdc++fs
< vasild>
indeed, and it looked like a small innocent change...
< vasild>
"nothing could go wrong with this"
< MarcoFalke>
Oh. The ci issues can probably be fixed by installing gcc 8 (it is available in bionic)
< vasild>
hmm
< vasild>
MarcoFalke: don't we want to stick to the default compiler for a given distro?
< vasild>
in order not to ask users to bother with installing a compiler?
< MarcoFalke>
unsure, maybe for the 0.22 release, but later we could change that
< vasild>
22.0 you mean? :)
< MarcoFalke>
heh
< sipa>
wumpus: ELF looks surprisingly simple
< dongcarl>
Posted some debugging notes for the llvm-nonreproducibility issue
< dongcarl>
In #bitcoin-builds
< wumpus>
sipa: it is!
< bitcoin-git>
[bitcoin] luke-jr closed pull request #19488: Refactor mempool.dat to be extensible, and store missing info (master...mempool_dat_extensible) https://github.com/bitcoin/bitcoin/pull/19488
< sipa>
wumpus: for the python typing annotation you want to run it in mypy
< wumpus>
right, I don't have that installed locally, I probably should
< luke-jr>
wumpus: thoughts on using 0.20.2 & 0.21.0 as a trial run for how fast users could upgrade for Taproot?
< luke-jr>
maybe release notes could include a short paragraph to this effect
< aj>
luke-jr: they're both important preparatory steps before deploying taproot (0.20.2 avoids network spam, 0.21.0 shakes out the major upgrade bugs), so not sure it's even a "trial" rather than the first leg of the relay or something?
< luke-jr>
aj: I mean in the sense of "If you want Taproot activated sooner, upgrade ASAP to show how quickly deployment can happen"
< aj>
luke-jr: yeah, i just mean that you should be upgrading to those versions for practical reasons anyway, not just because of how people might read the stats later
< luke-jr>
aj: perhaps, but without a kick like this, I'm not sure many people will see it that way
< luke-jr>
or might figure "I'll just wait and upgrade immediately when activation is released"
< TallTim>
lol I still run 0.13 on an old box
< TallTim>
newer on newer machines.
< luke-jr>
TallTim: I have 0.13 for my cold wallet too :p
< TallTim>
Armory pretty much requires older client as far as I know
< luke-jr>
:/
< TallTim>
its not maintained like it was, so its frozen in time
< TallTim>
I specify a data directory on my NAS for v0.19 to dump blockdata. I keep a monthly archive and an active blockchain directory.
< achow101>
luke-jr: fanquake: can you try rebuilding and clearing the cache and see if you get the other result?
< luke-jr>
I didn't have a cache to begin with…
< luke-jr>
aside from the dpkg cache, which I intentionally have set to never update
< achow101>
yeah I just want to see if repeated builds by the same builder get different results
< fjahr>
jonatack: I need to finish addressing john's feedback, on #19055. #19521 can be reviewed in parallel but I hope 19055 can be ready soon, that would make things easier.
< achow101>
jonasschnelli: does core-meetingbot have a command to do the channel ping?
< jonasschnelli>
achow101: not yet. That would be another plugin
< jonasschnelli>
I can try to add it.
< achow101>
it'd be super helpful
< jonasschnelli>
Agree.
< miketwenty1>
does anyone have stats / data on how many downloads or what the usage is of the different bitcoin node downloads? linux/windows/mac ?
< wumpus>
no, no one is keeping download statistics
< miketwenty1>
wumpus wouldn't the admins of bitcoin-core.org / bitcoin.org have these stats.. not sure what you mean by "no one"
< wumpus>
there are no trackers or anything on the site; it's sometimes inconvenient but for a project like this it's very important to stay above the table in that regard
< miketwenty1>
bitcoincore.org*
< miketwenty1>
i see.. im guessing you help run the downloads available on bitcoincore.org?
< achow101>
this is information that I will be gathering in my survey
< miketwenty1>
achow101: i look forward to seeing comparisons in percentage to downloads of (linux/windows/mac) .. specifically knowing how many mac downloads we have in comparison will be interesting to me
< bitcoin-git>
[bitcoin] theuni opened pull request #20436: depends: Add a nasty qt hack to work around clang non-determinism (0.21...fix-clang-qt-determinism) https://github.com/bitcoin/bitcoin/pull/20436
< bitcoin-git>
[bitcoin] practicalswift opened pull request #20437: fuzz: Avoid time-based "non-determinism" in fuzzing harnesses by using mocked GetTime() (master...fuzzers-remove-time-based-non-determinism) https://github.com/bitcoin/bitcoin/pull/20437