<glozow>
yeah. I was going to push my rebase but now I'm basically rewriting it 😅
<achow101>
do you want to put 28199 on the board instead of 28031?
<glozow>
oh yes please thanks
<glozow>
forgot to do that
<achow101>
don
<glozow>
thanks!
<achow101>
#topic BIP 324 updates (sipa)
preimage has joined #bitcoin-core-dev
<sipa>
hi
<sipa>
We're getting there.
<sipa>
There are currently 3 PRs open.
<sipa>
The ciphersuite one, the transport abstraction, and then the v2 transport itself
Chris_Stewart_5 has quit [Ping timeout: 252 seconds]
<sipa>
With the last one, you can experiment with bip324 connections, though in a test-only form only (using -bip324=ip).
<achow101>
ciphersuite seems to be getting review, looks close to merge
<darosior>
(hi)
<sipa>
bitcoin.sipa.be runs this code, if you need a remote to connect to (-addnode=NAME -bip324=NAME should suffice).
andrew_mo_ has quit [Ping timeout: 245 seconds]
<sipa>
Going forward, I think I'm going to wait for the ciphersuite and transport abstraction to make progress first.
Chris_Stewart_5 has joined #bitcoin-core-dev
<instagibbs>
session_id happens regardless of peer's behavior?
<instagibbs>
(assuming within the ranges specified)
<sipa>
After that, I'll probably drop the test-only -bip324 option from the v2 transport stuf, and implement actual service bit based decisions, maybe in a separate PR.
<sipa>
@instagibbs It appears as soon as we know the other side has the same key we do.
<instagibbs>
ill ask more offline :)
<sipa>
It's empty ("") otherwise, regardless of v1 or v2.
<sipa>
So if you see a non-empty session_id, the other side should show the same.
<MacroFake>
(For example it would allow to run all CI tasks on a riscv machine)
<achow101>
Or any other topics to discuss today?
<MacroFake>
Thanks
<sipa>
@MacroFake What is the status of CI, with cirrus changes?
<MacroFake>
sipa: I think hebasto is moving macOS and Windows to Github CI
<MacroFake>
The macOS seems ready, apart from the feedback I left
andrew_mo_ has joined #bitcoin-core-dev
<MacroFake>
Also, it would need a maintainer to enable Github CI (if we want to run it on pull requests)
<fanquake>
Yea the PR description in that PR needs updating to actually explain what’s its doing (dropping support for a release platform from CI), rather than making it look like just swapping infra
<sipa>
So will nothing be on cirrus anymore?
<dergoegge>
macOS and Windows on persistent workers is not an option?
<sipa>
Or only self-hosted things?
Guest35 has joined #bitcoin-core-dev
<MacroFake>
sipa: I think we'll keep Linux on Cirrus, but with persistent workers
<achow101>
aren't persistent workers self hosted?
<MacroFake>
yes, it is the same
<achow101>
who's hosting them?
<sipa>
Who is sponsoring those?
<MacroFake>
Sponsors exist
<sipa>
I guess I used the wrong terminology; they're running on our hardware, but it still goes through the Cirrus system for managing.
<sipa>
Right?
<MacroFake>
yeah, it is called cirrus-cli (or so)
<fanquake>
dergoegge: it’s not clear if that’s possible, or just hasn’t been investigated before picking GitHub CI
<sipa>
One question that came up in the libsecp meeting was what to do there, and if these cirrus persistent workers would also be available for libsecp256k1.
<MacroFake>
dergoegge: macOS is possible, but I don't think anyone wants to work on that. Do you?
<MacroFake>
sipa: They will
<dergoegge>
MacroFake: no
<MacroFake>
dergoegge: heh
<sipa>
Ideally libsecp can follow whatever Bitcoin Core does, but only if that doesn't add costs/burden to people running it.
<MacroFake>
sipa: Will libsecp use Github CI for Windows and macOS?
<sipa>
I guess we'll have to.
<luke-jr>
does this break CI for non-Core repos?
andrew_mo_ has quit [Ping timeout: 260 seconds]
<MacroFake>
luke-jr: If you want to use Github CI, you'll have to enable it
<dergoegge>
It would be good to have a migration plan, should the current host of the persistent workers no longer be willing to host
<luke-jr>
I was thinking the persistent-workers thing
<achow101>
persistent workers don't work for forks
<MacroFake>
luke-jr: Ah, if you want to use a persistent worker, you'll have to add it to your Cirrus organization
<MacroFake>
luke-jr: An alternative would be to create a pull request to use Github CI Linux boxes, where possible
<fanquake>
I guess libsecp will also set up/run its own persistent workers for Linux too?
<luke-jr>
MacroFake: does it no longer work with normal Cirrus CI workers?
<MacroFake>
(but it is not possible for all tasks)
<sipa>
I'm not super happy about relying (even) more on GitHub for infrastructure, but as Cirrus basically only works for GitHub anyway, it's not much of a change, and it seems there really aren't many alternatives (except very expensive ones).
<achow101>
luke-jr: I think cirrus will cut you off or otherwise limit your ci usage
<MacroFake>
luke-jr: That would work, with conditional tasks
<MacroFake>
You have 40$/mo free capacity
<MacroFake>
If you only do 4 pushes per month, it should be enough
<MacroFake>
But I am not sure if this is something we need/should support
<achow101>
it's nice for people to be able to run ci on their own branches before opening a pr
<MacroFake>
achow101: FWIW, I have CI disabled on my branches in my fork
<sipa>
Is setting up a (linux) CI persistent worker hard? If not, we can probably manage our own for libsecp.
<achow101>
MacroFake: what's the reason for not migrating the linux tasks to github ci?
<luke-jr>
achow101: yes, especially if we don't want people pushing to PRs for testing
<MacroFake>
achow101: For some it is not possible
<MacroFake>
achow101: For others: No one has opened a pull request to do it yet
<luke-jr>
(IIRC someone was asked not to do that recently)
<fanquake>
that’s mostly only when it’s clear the person isn’t even bothering to compile locally
<MacroFake>
sipa: It is one line of bash: "cirrus worker run --labels type=lunar --token todo_get_the_correct_token"
<fanquake>
and just keeps for pushing the the PR to see what works
<sipa>
FWIW, I personally use CI to run tests for my own PRs, for everything beyond what make check / test_runner doesn't do (sanitizers, linters, ...).
<MacroFake>
)
<MacroFake>
I think it is fine to use CI for debugging, as long as you do a local make check && test_runner.py
<luke-jr>
in any case, it would be easier for people to do that if they don't need to spam PRs with updates ;)
<sipa>
I'd like to run linters locally too, but haven't actually ever looked into what is needed for that.
<MacroFake>
sipa: You'll need docker for the linters, but I don't run the either locally
<MacroFake>
You can push and get a result from CI in 120 seconds or less
<fanquake>
you can also skip Docker and use a venv, and just run them directly
<sipa>
Right.
<MacroFake>
fanquake: I wouldn't trust them. They are random pip packages from the internet and even raw x86 binaries (shellcheck)
<cfields>
I think we're in kinda a bad state if most day-to-day devs aren't running these locally :\
<cfields>
local-first tests used to be an explicit goal
<MacroFake>
(Ok, I guess docker doesn't help if they are viruses)
<cfields>
(I'm guilty as well)
<josie>
also guilty of using CI to run the linters / sanitizers. been meaning to look into what it would take to run all this locally, but last time I tried, it wasn't super straightforward
<fanquake>
cfields: unit and functional tests should be run locally. Linters and cross-compilation (CI) I’m sure are much less likely
<MacroFake>
cfields: There is some stuff that just can't be run locally. Are you going to compile santizers+fuzz+clang-tidy for every commit locally?
<sipa>
@cfields I agree it'd be better if more we easily/automatically doable locally.
<sipa>
But also, if it's just linters, there isn't much trust involved.
andrew_mo_ has joined #bitcoin-core-dev
<fanquake>
I think it’s also a case of time/resource restriction. With Docker, running any of the CI jobs locally is very straightforward
<MacroFake>
If you trust docker, all you need to run is "DOCKER_BUILDKIT=1 docker build -t bitcoin-linter --file "./ci/lint_imagefile" ./ && docker run --rm -v $(pwd):/bitcoin -it bitcoin-linter"
<cfields>
MacroFake: I think it should be _possible_ for me to do all of that stuff on a commit that I consider scary and worthy of testing before pushing. Of course not for every commit.
<sipa>
I should learn docker.
<MacroFake>
cfields: It is possible
<cfields>
(And of course it's possible. It's just currently cumbersome, which is the topic)
<fanquake>
cfields: definitely entirely possible to do that locally
<fanquake>
cumbersome at this point is install Docker and run 1 bash script
<MacroFake>
cfields: I don't think it can be made easier or take less time. Compiling takes time
<josie>
fanquake: the struggle for me was figuring out how to run *just* the jobs I want e.g. linting
<MacroFake>
sipa: I mean with trust that the linters can corrupt your .git folder, for example
<josie>
I'm sure it just involves some docker-fu involved, but I couldn't figure it out last time I tried. A few examples in the docs would probably go a long way
<fanquake>
josie: the linters are a bit of an outlier yea, for all other CI jobs, you just pick the one you want, and call the bash script (with Docker installed) and it'll ru
<cfields>
Ok, ranting is beyond the scope here. But personally, I'm hoping after CMake we can get some more checks integrated into the buildsystem for easier testing. Multi-config builds for example, would allow for sanitizer builds in parallel.
<MacroFake>
josie: It is a single line of bash: "DOCKER_BUILDKIT=1 docker build -t bitcoin-linter --file "./ci/lint_imagefile" ./ && docker run --rm -v $(pwd):/bitcoin -it bitcoin-linter"
<MacroFake>
(taken from the docs)
<josie>
MacroFake: ah, nice, I haven't checked the docs recently (or might have just missed this before)
<fanquake>
cfields: I can't wait for CMake to magically give everyone more time and CPU heh
<sipa>
@cfields That sounds great.
<MacroFake>
I can convert the linter to the same syntax as the other tests, but probably not before next month
<sipa>
(even if just aspirational)
<cfields>
fanquake: for me: less docker == more time. Maybe I'm just old :)
<josie>
next morning pizza is the best pizza.. its never too early for pizza
<luke-jr>
achow101: how did we end up with such a meeting time? XD
<MacroFake>
cfields: do they? It should be optional, right
brunoerg has quit [Remote host closed the connection]
<cfields>
sipa: I wonder if we could create a new virtual environment to virtualize them all...?
<cfields>
it's probably never been tried.
<MacroFake>
fwiw, I usually run docker/podman in a VM
<cfields>
MacroFake: I was mostly joking. Just another tool to have hooked up :)
<luke-jr>
I don't really trust VMs much post-Spectre :/
<MacroFake>
(vm on a remote machine)
<luke-jr>
that's better :)
<MacroFake>
But I don't trust ssh :(
<luke-jr>
considering the recent ssh client vulnerability, I can't blame you :|
andrew_mo_ has joined #bitcoin-core-dev
andrew_m_ has joined #bitcoin-core-dev
andrew_mo_ has quit [Ping timeout: 246 seconds]
Guest35 has quit [Quit: Client closed]
andrew_m_ has quit [Ping timeout: 246 seconds]
<sipa>
@[luke-jr] Why does Spectre matter? That may just leak private information to software running on the same machine, but in A CI setting I hope there is no private information.
brunoerg has joined #bitcoin-core-dev
andrew_mo_ has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 244 seconds]
<vasild>
I also use CI on my own repo to debug CI failures that only happen on CI and I can't repro locally (simplest case - compile failure only on windows)
brunoerg has joined #bitcoin-core-dev
<vasild>
fwiw in MySQL development there were a few test branches that had full CI hooked on them and everybody was "push -f"ing to them
andrew_mo_ has quit [Ping timeout: 260 seconds]
<sipa>
@Sjors What did you set for -bip324= ?
<sipa>
Interesting; that must mean either some other people are running it with *, or are running the old bip324 branch.
brunoerg has quit [Ping timeout: 260 seconds]
andrew_mo_ has joined #bitcoin-core-dev
puchka has quit [Ping timeout: 246 seconds]
andrew_m_ has joined #bitcoin-core-dev
andrew_mo_ has quit [Ping timeout: 246 seconds]
puchka has joined #bitcoin-core-dev
andrew_m_ has quit [Ping timeout: 260 seconds]
andrew_mo_ has joined #bitcoin-core-dev
andrew_m_ has joined #bitcoin-core-dev
andrew_mo_ has quit [Ping timeout: 250 seconds]
andrew_m_ has quit [Ping timeout: 250 seconds]
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 258 seconds]
brunoerg has joined #bitcoin-core-dev
andrew_mo_ has joined #bitcoin-core-dev
<luke-jr>
sipa: well, this is why I have my development machine on separate hardware - but if it was all on teh same machine, the CI could potentially extract sensitive info from the host
brunoerg has quit [Ping timeout: 260 seconds]
andrew_mo_ has quit [Remote host closed the connection]
andrew_mo_ has joined #bitcoin-core-dev
dgenr8 has joined #bitcoin-core-dev
brunoerg has joined #bitcoin-core-dev
andrew_mo_ has quit [Remote host closed the connection]
andrew_mo_ has joined #bitcoin-core-dev
andrew_mo_ has quit [Ping timeout: 246 seconds]
andrew_m_ has joined #bitcoin-core-dev
Guest35 has joined #bitcoin-core-dev
Guest7 has joined #bitcoin-core-dev
andrew_mo_ has joined #bitcoin-core-dev
andrew_m_ has quit [Ping timeout: 246 seconds]
jespada has quit [Read error: Connection reset by peer]
andrew_mo_ has quit [Ping timeout: 260 seconds]
jespada has joined #bitcoin-core-dev
andrew_mo_ has joined #bitcoin-core-dev
andrew_mo_ has quit [Ping timeout: 252 seconds]
andrew_mo_ has joined #bitcoin-core-dev
andrew_mo_ has quit [Ping timeout: 246 seconds]
Talkless has joined #bitcoin-core-dev
andrew_mo_ has joined #bitcoin-core-dev
andrew_mo_ has quit [Ping timeout: 260 seconds]
andrew_mo_ has joined #bitcoin-core-dev
andrew_mo_ has quit [Ping timeout: 264 seconds]
andrew_mo_ has joined #bitcoin-core-dev
andrew_mo_ has quit [Ping timeout: 246 seconds]
<jonatack1>
Yes, the docker script in test/lint/README.md works for me, i.e. for the first run: