<stickies-v>
#topic Benchmarking WG Update (l0rinc, andrewtoth)
<l0rinc>
Recently I've been focusing on finding accidental quadratic algos throughout the codebase (after bumping into one in #35889), had a few good findings
<l0rinc>
rebasing and reviewing similar findings, that's it from me
<andrewtoth>
nothing from me
<stickies-v>
#topic QML GUI WG Update (johnny9dev)
Murch[m] has quit [Changing host]
Murch[m] has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
<johnny9dev>
We have a new contributor to the project, uqlidi, who has been helping fix up bugs and compatibility issues with the qml. pseudoramdom has been doing a lot of good work addressing feedback on the preview release and making really nice design updates to all of the views. epicleafies has also been addressing feedback on the Acitivity page
<johnny9dev>
I made a good number of updates to the first staging branch pr (bitcoin-core/gui-qml#871)
<bitcoin-git>
bitcoin/master d564b02 Hennadii Stepanov: iwyu: Add temporary mapping to work around upstream bug
<bitcoin-git>
bitcoin/master 204256c merge-script: Merge bitcoin/bitcoin#35900: iwyu: Fix warnings in `src/interfaces` and tr...
<bitcoin-git>
bitcoin/master b3ff9c4 Hennadii Stepanov: iwyu: Fix warnings in `src/interfaces` and treat them as errors
<bitcoin-git>
[bitcoin] fanquake merged pull request #35900: iwyu: Fix warnings in `src/interfaces` and treat them as errors (master...260805-iwyu-interfaces) https://github.com/bitcoin/bitcoin/pull/35900
<bitcoin-git>
[bitcoin] purpleKarrot opened pull request #36101: doc: Add an error handling strategy (master...error-handling-strategy) https://github.com/bitcoin/bitcoin/pull/36101
<achow101>
hi
<johnny9dev>
as a reminder, the staging branch is what we hope to merge into bitcoin/bitcoin
winterrdog has quit [Read error: Connection reset by peer]
<johnny9dev>
I got some really good feedback on it already and have shifted some things around. One of the bigger changes will be balancing our test automation strategy by including an integration suite that run QtTest to test the Gui+Node in process
<hebasto>
asking for more eyes in qml#881 as it paves the way for further work on the staging branch
<corebot>
hebasto: Error: That URL raised <HTTP Error 404: Not Found>
<Murch[m]>
It’s great to see the progress y’all are making. Maybe we can do another testing party when you think it’s in a good place for that
<johnny9dev>
this will let us move a lot of our end to end functional regression tests into a c++ qttest target
<johnny9dev>
I'm iterating on the description to try to make it clear what the decisions currently made are and I hope that the current description should at least make it clear the main things the PR does to create the initial foundation
jonatack has joined #bitcoin-core-dev
<hodlinator>
hi
<jonatack>
hi
jerryf has joined #bitcoin-core-dev
<johnny9dev>
but yeah looking for feedback and opinions on the staging PR now as it will have a significant impact on the staging going forward.
jerryf_ has joined #bitcoin-core-dev
<johnny9dev>
and everything learned from the foundation staging PR i will be working to add to our development qt6 branch
<johnny9dev>
that is all
jerryf has quit [Remote host closed the connection]
<stickies-v>
#topic Bitcoin Core and AI usage, perception and reality (Moneyball)
<Moneyball>
Several people, including donors have approached me expressing concern that Bitcoin Core devs "aren't paranoid enough about AI cyber threats" and that the Core project isn't sufficiently leveraging AI tools for tasks like assisting issue and PR triage, code review, and test case development.
<Moneyball>
I've spoken to a few Core devs, so I am aware of some AI activity and usage within Core, but to be honest I don't have a clear sense of the extent of its use. I sense that it is used more than the current perception suggests. It is also my sense that its usage could increase further. In any case, I don't really have anything substantial to share with folks.
<Moneyball>
Thoughts on publishing more about how individuals or the project as a whole is utilizing AI for offense and defense?
satsfy_ has quit [Ping timeout: 242 seconds]
<pinheadmz>
We're all using Claude lol
<pinheadmz>
There's a few engineers like furszy building their own harness and scanning for things 24/7 or however many tokens he gets per week
jerryf_ has quit [Remote host closed the connection]
<pinheadmz>
Not to mention Rob Ham and "red team"
jerryf has joined #bitcoin-core-dev
<achow101>
plenty of people are using ai for assistance. but just because it exists also does not mean that everyone needs to use it
<yuvicc>
project loupe as well ig
<stickies-v>
I use LLMs quite heavily for additional review. I'll do my own manual review, have an LLM do its own parallel pass, and then combine both / dive deeper from there
<achow101>
and frankly, ai use has produced so much noise and garbage that i'm not interested in seeing more from people who don't know what they're doing
<Moneyball>
I think that attitude is the problem
<stickies-v>
but yeah I prefer people owning their reviews instead of fully automated AI reports which seem to be very very noisy on our codebase
<Moneyball>
I'm well aware of where AI is and is not valuable.
<eugenesiegel>
I am not involved in red team or project loupe or anything and i can't speak for the project as a whole, but I don't use claude that much except for dumb scripts. I also think a lot more effort could be spent training people to actually spot bug classes instead of relying so much on AI but that's probably a whole conversation. I'd also be worried
<eugenesiegel>
about this encouraging or forcing contributors to use AI and I can think of another project that uses it really heavily and I honestly think the quality of the codebase has gone down since. Probably have more opinions that can't fit in here, and I'm not an ai doomer or anything...
<Murch[m]>
My impression is that many or most of the Bitcoin Core contributors have started incorporating AI tooling into their workflows and given their prior expertise get great use out of it.
<Moneyball>
My high level point is "how does the bitcoin ecosystem know what the Core project is and is not doing with respect to AI in order to reassure folks that the contributors to the project aren't sticking their heads in the sand"
<dzxzg>
So a broad spectrum of views and ways that people are using it
<Moneyball>
I'm here to help. I'm aware of several efforts. Not all. But I don't really know what to tell people who ask.
<achow101>
The "project" is not using it in any way. individuals are using ai in a variety of ways
<sedited>
Moneyball, I'm not sure where the impression of sticking their heads in the send comes from.
<jonatack>
Moneyball: from where do these folks get their impressions
<eugenesiegel>
I kind of equate ai scanning to something like sqlmap where you scan for low hanging fruit (i.e. common bug classes like reading past end of buffer, etc) and more complicated bugs are probably not going to be found with it (yet).
<andrewtoth>
I use AI tools for absolutely everything. I don't think I can go back to not being able to use it. I never post text I expect other humans to read from AI though, I always rewrite in my own words so I am sure I understand what I'm writing.
<Moneyball>
yes achow101 that is precisely why i said contributors
<Moneyball>
as i knew you'd have snide remarks
<sedited>
We're frustrated with slop contributions, yes, but I think many of us are using the tools pretty effectively.
<andrewtoth>
eugenesiegel: I see the opposite. It's finding obscure bugs that would be very difficult for humans to find.
<Moneyball>
jonatack i'm not sure where people get their impressions but wherever they search or look i don't think they'd find the answer to the question
<eugenesiegel>
andrewtoth: where? in bitcoin core or somewhere else?
jerryf has quit [Remote host closed the connection]
winterrdog has joined #bitcoin-core-dev
<andrewtoth>
clightning is one example, I haven't seen the bugs that were found though
<johnny9dev>
What can be done even? As these things seem to get better it doesnt look like anything other than people being even more productive?
<Moneyball>
a group of us are meeting in a couple of weeks to discuss learnings from Loupe and red team, how projects can handle triage of these reports, and we will be publishing best practices. this will include l0rinc and justin from Localhost
<lightlike>
there are different ways: a few people use AI to find ideas for issues/PRs (I don't), more people (including me) use it as a tool in PRs / issues they've come up with by themselves.
winterrdog has quit [Read error: Connection reset by peer]
<eugenesiegel>
andrewtoth: I haven't really seen an example of something that's so complicated a human couldn't find it, but if I'm sticking my head in the sand I'd like to know. May be anecdotal, but I haven't seen nearly the same amount of attention given to something like syzkaller in the past (linux kernel fuzzer) compared to the attention AI is getting now
<andrewtoth>
what about the zcash inflation bug?
<Moneyball>
eugenesiegal i can definitely state that Loupe has found many, many vulns across dozens of projects. some are high severity. many are medium severity but these can be chained together to create a high severity outcome
antanst71255 has quit [Ping timeout: 258 seconds]
<eugenesiegel>
andrewtoth: I don't know much about the zcash bug, so I'm probably out of the loop
<johnny9dev>
Even if a human can find it the speed and thoroughness of the frontier models are undeniable
<andrewtoth>
Moneyball: if the AI can't chain the medium severity bugs itself to produce the high severity, then I find these claims kind of hand-wavy
<eugenesiegel>
Moneyball: speaking from experience, some of the codebases I've seen in the space are really lacking on the security side and are missing really basic checks. but i'll wait for public disclosure to judge for myself
<sedited>
Moneyball, "issues as PRs" has not worked well here so far. Many bots do take things off the issue tracker, but a lot of them produce slop that then needs more triaging and work from more people. Take this PR from today for example, which took away attention from a bunch of us, but didn't do anything to move things forward: https://github.com/bitcoin/bitcoin/pull/36055 . Can obviously say
<sedited>
that the problem is the issue tracker, but would take significant reform to change.
<dergoegge>
Personally I think core has been holding up well against the "AI threat" because we have very strong testing for most important things, that other project like e.g. some of the lightning impls have not invested in the prior years
<sedited>
dergoegge +1
<jonatack>
I use free AI sometimes to verify things. I don't use it at all (for now) for writing. I would like to ramp up on open source models but in El Salvador it is hard to obtain hardware without importing it yourself. Frontier token use is a cost I believe I can't take on alone.
<jonatack>
dergoegge: agree
jerryf_ has joined #bitcoin-core-dev
<johnny9dev>
yes bitcoin's strong engineering fundamentals are paying off
<dergoegge>
That said I do think AI vuln scanning is obviously a part of any project that wants to take security seriously
jerryf has quit [Remote host closed the connection]
<jonatack>
yes
<Moneyball>
sedited i'd be happy to engage you and interested folks on exploring how to handle AI noise. can follow up after
<l0rinc>
I started scanning for bugs and either fixing them or reporting them to security list (to Core and a few other projects) for the past few months. Unfortunately this is often costing me thousands of dollars per day so far but OpenSats just approved a mini-grant that I could use to continue these scans.
<l0rinc>
I understand that some people think the findings are garbage - which was the case up until a few months ago -, but that's not the case anymore, basically all of the findings are real now, but without the cyber capabilities only the low-severity ones gets surfaced.
<eugenesiegel>
andrewtoth: thanks for the link. I just don't know how to evaluate this, so I don't really know how complicated it is. Though there's something to be said that it wasn't found prior and was only found via AI
satsfy_ has joined #bitcoin-core-dev
<achow101>
Moneyball: I don't understand what the concern in your original question is. Do people think that ai is not being used?
<Moneyball>
also the question of "how is Core utilizing AI" isn't just about AI security scanning but also other use cases of AI
<Moneyball>
achow101 people have a belief the Core project is behind the curve. I don't have anything to point them to that counters that view
<tigerMafia>
Moneyball: about mindless AI PRs (slop) - i agree with achow101, if someone is does not know what they're doing, LLMs will just amplify that skill issue a lot more and make it a pain for the reviewers. personally, i have been faced with fully AI PRs where authors bank on reviewers to steer their harness (not so good). if someone copy-pastes AI output, bit for bit, i ignore the PR.
<maxedw>
I got to see Furzy's harness in action and it was quite impressive, I know he has been scanning the codebase for months. If external people want to help in those efforts he could probably massively benefit from access to the most capable model without the security classifiers.
winterrdog has joined #bitcoin-core-dev
<Moneyball>
tigerMafia i fully appreciate that and sympathize with you. we're seeing it across many projects and as mentioned there are various attempts to combating it.
Rob1Ham has joined #bitcoin-core-dev
jerryf_ has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
<pseudoramdom>
How are other projects giving the perception of "using AI" apart from just scanning issues?
winterrdog has quit [Read error: Connection reset by peer]
<Murch[m]>
My take away is that many contributors have been using AI, the project has stood up well to recent red team probing, but AI tools have not been incorporated at the project level yet. Presumably the absence of the latter is what prompts the concern
<dergoegge>
Not to jinx it but, maybe the fact that the bitcoin network is still up and running, is a hint that core is not behind on things?
<jonatack>
Moneyball: AI has been (over-)used to prompt finished-looking, verbose, often nonsensical, techno-babbly BIP drafts by nyms that (often) open them directly as PRs to the BIPs repo. OTOH, it is being used well by established bitcoin protocol devs to propose useful BIPs.
<sedited>
sounds like the problem is more that people are not blogging enough :P
<Murch[m]>
pseudoramdom: E.g., automatic review of new issues and PRs by agents
<achow101>
perhaps people are more interested in doing useful things with ai rathe than talking about it
winterrdog has joined #bitcoin-core-dev
<andrewtoth>
I would not be happy if we had a copilot-like bot that posted automatic review comments, if that is an idea someone would have
<l0rinc>
andrewtoth: copilot is not very smart, but making drahtbot smarter to find non-surface level findings would be very useful
<yancy>
Perhaps part of the perception issue is also that core is a C++ code base, and AI is not maybe as effective at writing C++ as it is Rust due to the type system. Just speculating why perception is different from the reality.
<johnny9dev>
It does almost feel too early to add anything additional to bitcoin core. On top of it, bitcoin core is likely extremely well understood already by llms so is there any markdown file that can even really add any additional value to this?
<l0rinc>
yancy: LLMs are very proficient in C++ now
<johnny9dev>
codex is very good at modern c++
<andrewtoth>
l0rinc: I don't want automatic comments from bots. If a reviewer uses a smart AI and then triages the results in their own words, ensuring no false positives, that would be welcome.
<tigerMafia>
andrewtoth: +1
<lightlike>
agree, AI during review is most helpful if you can interact with it / steer it into directions where real issues may be. an automatic AI review isn't helpful, everyone can do a rough analysis locally and then take it from there.
<andrewtoth>
I can run my own AI reviewer myself, I don't need it posting on my PR
<l0rinc>
andrewtoth: I sympathise with the feeling, but some PRs have so many bugs that I would prefer an AI point them out so by the time we get to it the trivial ones are already handled
winterrdog has quit [Read error: Connection reset by peer]
<andrewtoth>
no
<l0rinc>
yes :)
<eugenesiegel>
are there certain areas that LLMs are lacking in or are security engineers cooked?
<achow101>
i absolutely do not want automated review
<maxedw>
also not a fan of automated review, I use privately and find 4/5 complaints are non-issues or random opinion
<tigerMafia>
andrewtoth: +1. already github has an issue with handling comments.. now imagine the volume of comments that would be made by copilot
jerryf_ has joined #bitcoin-core-dev
Rob1Ham_ has joined #bitcoin-core-dev
<Moneyball>
eugenesiegel there is still value in the human driving the LLM to find security vulns. being a subject matter expert helps.
Rob1Ham_ has left #bitcoin-core-dev [#bitcoin-core-dev]
<l0rinc>
I also don't like spammy comments, but if configured well, an automated AI review would be extremely useful. We all have PTSD from last year's models which were mostly noise: that's not the case anymore.
<Murch[m]>
Well, seeing the garbage we get in Optech and the BIPs repository, it doesn’t seem everyone has stopped using last year’s models
<jonatack>
:D
winterrdog has joined #bitcoin-core-dev
<_aj_>
have individual contributors run automatied AI reviews, read/understand them, then pass on the key findings without huge chunks of AI claude-ish verbiage
<eugenesiegel>
do people have certain prompts they use for this? Even when I ask claude benign security-related questions, it refuses to respond. And I haven't gotten access to the chinese models yet :/
jerryf has quit [Ping timeout: 248 seconds]
<tigerMafia>
eugenesiegel: chinese models tend to respond, no qualms
<andrewtoth>
_aj_: +1
winterrdog has quit [Read error: Connection reset by peer]
winterrdog has joined #bitcoin-core-dev
<janb84>
_aj_: for now the best strategy I think
<l0rinc>
_aj_: absolutely, but that's a lot of work, it shouldn't be the only way to receive feedback - this way I personally just ignore reviewing PRs that aren't structured well. Maybe we could enable AI review for new contributors only and see how well it performs.
<fanquake>
_aj_: mixed output on that front. I think we still need to push for better understanding of issues before they are being forwarded, along with the production of accompanying (functional) tests
jerryf_ has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
<andrewtoth>
the "read/understand them" part is the hardest
<andrewtoth>
otherwise you offload that onto the author, rather than that being the reviewer's job
<l0rinc>
if the AI only receives a patch (without any tool calling and full repo access) we can't expect depth. But a proper review bot that checks out the change and does its thing is extremely capable lately!
<willcl-ark>
people are worried we don't have enough AI in the Github repo?
<tigerMafia>
:)
<andrewtoth>
l0rinc: we can all do that locally though, no need to pollute github with it
<l0rinc>
we can, but most people aren't and we're commenting on stuff that an AI could have found
<Moneyball>
willcl-ark not sure if that is a joke or actual question
<achow101>
l0rinc: so?
<willcl-ark>
Moneyball: I'm catching up from "donors have approached me expressing concern that Bitcoin Core devs "aren't paranoid enough about AI cyber threats" and that the Core project isn't sufficiently leveraging AI tools for tasks like assisting issue and PR triage, code review, and test case development."
winterrdog has quit [Read error: Connection reset by peer]
winterrdog has joined #bitcoin-core-dev
winterrdog has quit [Client Quit]
<l0rinc>
ahow101: so if human review is the bottleneck, let's use AIs to help with the low hanging comments
<achow101>
ai found an issue, human finds an issue, same result, issue is found and fixed
<achow101>
like if i'm reviewing a pr, and someone else reviews the same pr, and we both find the same issue, that's not a problem. don't see why it would be any different if an ai did it or not
<l0rinc>
achow101: no, it's like proofreading a document and you find typos - a spell checker should have caught those instead of wasting human reviewer efforts
Rob1Ham has quit [Quit: Client closed]
Rob1Ham_ has joined #bitcoin-core-dev
jerryf_ has joined #bitcoin-core-dev
<_aj_>
l0rinc: same applies to a linter or running CI; in this case the idea is that they're not quite reliable enough to run as part of CI on every PR, so you should run them yourself as an author or reviewer
<achow101>
i don't think it's wasting, but i get your point
<eugenesiegel>
do people have any tips to make the output of the llms less noisy or to focus on things that are actionable?
jerryf has quit [Remote host closed the connection]
<Rob1Ham_>
Hello everyone, first time using IRC in 20 years, I was told to join, happy to help however I can.
Rob1Ham_ is now known as Rob1Ham
<willcl-ark>
AFAIK ~ everyone is using AI tooling locally to assist with coding, review, building tooling, benchmarking, test case development, fuzzing, whatever else they're working on. The only thing "missing" for these worried people therefore is the "lack" of automated bot review in Github?
<johnny9dev>
eugenesiegel: not joking, tell it thats what you want
<l0rinc>
_aj_: "they're not quite reliable enough to run as part of CI on every PR" - I'd argue they absolutely are reliable enough now, given proper prompting and context and price
<_aj_>
l0rinc: other people don't agree, and arguing isn't going to convince them
<Murch[m]>
willcl-ark: That seems to be the takeaway
<Moneyball>
willcl-ark no one said anything about automated bot review in github as what is lacking in public communications
<jonatack>
willcl-ark: could also use hardware or token budgets, but maybe that's only me
<Moneyball>
murch how is that the takeaway? it seems made up? i never said that
<andrewtoth>
Can we just have a guideline to run your code through an LLM before posting? Nobody wants to read the output of an LLM on your PR, they want humans to write and respond to.
<willcl-ark>
OK perhaps I'm missing what is being looked for here. Publishing which tools we use and how?
<johnny9dev>
can't someone just go and try to secretly prove out an AI review bot?
<Murch[m]>
“the Core project isn't sufficiently leveraging AI tools for tasks like assisting issue and PR triage, code review, and test case development.”
<sipa>
i'm not sure what the point of this discussion is
<Moneyball>
murch that statement doesn't imply or state that auto review bots in github is required?
<Murch[m]>
Well, if you have a different take away, how about you share yours
<jonatack>
Moneyball: you would like info to point to, that can reassure worried people IIUC
<willcl-ark>
jonatack: I mean, I wouldn't be opposed to a dev token fund; then people could see how many token we were using!
<stickies-v>
we're also coming up on time. I suggest we wrap this up and if there are follow up specifics to be discussed after today's initial discussion, we can do that next week?
<dergoegge>
I think the takeaway is that we should be communicating our work more (which we discussed at prior coredevs as well)
<lightlike>
Moneyball: then what do you suggest should change from the status quo, where already almost everyone uses AI locally, with a few projects doing systematic bug finding? What would be "state of the art" if the status quo isnt?
<sipa>
Moneyball: are you asking what people are using? trying to encourage some kind of usage? better documenting?
<Moneyball>
sipa i was asked to come here to raise this topic. i'm merely asking "what content can be shared with inquiring minds about how the Core project isn't behind the times on utilizing AI." i have never brought up the topic of "you must use a review bot in github" that is made up.
<andrewtoth>
that was me bringing up it was a bad idea, sorry for the noise
<Moneyball>
andrewtoth no worries
jerryf_ has quit [Remote host closed the connection]
<willcl-ark>
I think that's the main way one would see issue and pr triage/review taking place in the open by AI
<achow101>
Moneyball: then this discussion should suffice as public documentation that it's happening
jerryf has joined #bitcoin-core-dev
<Rob1Ham>
I have plenty of thoughts on the topic, but don't wish to further derail the conversation if it is out of scope for this discussion. I'm happy to share my perspective at another time or individually, anyone is welcome to reach out to me anytime.
<Murch[m]>
Moneyball: your initial topic prompt specifically stated that people had the impression that we were behind the curve on AI, because “the Core project isn't sufficiently leveraging AI tools for tasks like assisting issue and PR triage, code review, and test case development.”, so it’s confusing to me that you push back on me concluding that someone looked at our repository and found it strange that we don’t have automated PR review.
<jonatack>
Rob1Ham: after the meeting can work, too, if people aren't against
<BlueMatt>
speaking on the LDK end, where we have a project that takes security very seriously but is also a rather complicated set of code - we get very high value with having a cheap opus review bot do a first pass on PRs. It regularly finds real issues, and that's a *really* dumb bot, people doing manual review passes with better LLMs obviously finds much more. On the security side, LLMs are better at finding almost any class of bug than humans at
<BlueMatt>
this point, there's almost no question, and if its not true give it about two months.
<Moneyball>
achow101 i can summarize this discussion and post about it or speak about it. i'd have many more detailed questions though. i could write these questions down and share. then if any Core dev is so inclined, you could answer them in a post or here or privately with me and let me know what you are comfortable with sharing publicly
<BlueMatt[m]>
there's a lot of value in being a subject-matter expert and driving it in the right direction
<Rob1Ham>
My very brief summary is that the past month, of using either the frontier labs with cyber approved access, or the most recent open source models have seen a considerable step function in capability in finding vulnerabilites. I've used these tools for years and the past few months is a serious step fucntion.
<l0rinc>
BlueMatt: +1
<Moneyball>
murch sorry i just don't see how my statement in any way asserts auto bot github activity. it is made up.
<BlueMatt[m]>
LLMs are also at the point where I have very little desire to review human-authored code - LLMs are just better.
<dergoegge>
I had a similar discussion with Steve somewhere else, and suggested that he bring this up here. Specifically the public comms aspect.
<dergoegge>
Even though this may have derailed at the end here, I think there were some good points being made, and I'm sure this discussion can continue at a future coredev
<dergoegge>
Thank you, moneyball!
<achow101>
Moneyball: please do that
<BlueMatt[m]>
of course I also have no desire to review purely LLM-authored code where a human wasn't involved in taking a lot of care with how the code gets structured
<Moneyball>
achow101 will do. thank you.
<l0rinc>
moneyball: thanks for bringing up the topic
<stickies-v>
thanks for the discussion everyone, let's wrap it up here
<stickies-v>
#endmeeting
<corebot>
stickies-v: Meeting ended at 2026-08-27T17:05+0000
<Rob1Ham>
jonatack: happy to either after this or at a later time, my goal is to take the learnings and even the harness and make those public after some time as well so others can improve on it and make it work for their specific needs
jerryf has quit [Remote host closed the connection]
<johnny9dev>
I am personally very curious about when building a harness was very valuable. My experience has been each time I build one, the new model just does all of what i needed already so I just wasted a lot of time when I could have been focused on my work and waiting for the new model
jerryf has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 246 seconds]
jonatack has joined #bitcoin-core-dev
<johnny9dev>
or an example of a good working review harness would be interesting
<Rob1Ham>
johnny9dev: I've found the harness to be a very signifigant differentiator in the quality of the outputs, in being able to direct certain patterns/behaviors in finding issues
dzxzg has quit [Quit: onversation terminated!]
<Rob1Ham>
I've also been able to run circles out of claude/chatgpt with my harness using kimi k3/deepseekv4pro glm 5.3 etc
<Rob1Ham>
A harness also has dials that allow you to deal with precision and recall of scans, meaning do you want to only surface 100% true confirmed positive issues, or just try to get all things that are plausibly wrong with a code base, so you can use that dialing and fine tuning for different parts of the software development lifecycle based on the auidence consuming it and the intent of the
<Rob1Ham>
review.
jadi has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
Guest18 has joined #bitcoin-core-dev
<johnny9dev>
Are you able to share your harness publically?
<Rob1Ham>
johnny9dev: My goal is to do so! I'm trying to clean up the user experience and make it easy to configure all of the kinds of scans.
<Rob1Ham>
My inital harness was built in 30 minutes using kimi k3 in opencode to set up several loops of issues
jerryf has quit [Remote host closed the connection]
<Rob1Ham>
which was a step up in performance when compared to just using a single session window
jerryf has joined #bitcoin-core-dev
<Rob1Ham>
The intention is to have it tested among codebases i've scanned previoulsy, and use that for feedback and giving maintainers a time to review stuff before making it generally available, and its configurable so you can switch out models however you want.
<Rob1Ham>
and open source so other projects or teams can modify it to meet their own needs, or just take the concepts and have people reimpliment it however they wish
<johnny9dev>
Meaning instead of doing a full scan you have different prompts run separately for targeted classes of issues?
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
<Rob1Ham>
johnny9dev: exactly, I have a general quick scan looking at targetted issues such as entropy, key misamangemnet, supplychain issues, possible loss of fund codepaths etc, versus ones that are more open ended to let it be creative
<Rob1Ham>
It also somewhat turns into a bespoke harness for each codebase as it does more scans over time to identify issues
jerryf has joined #bitcoin-core-dev
<Rob1Ham>
I've also done stuff like built a library hook, where it will scan a codebase, see which cryptographic concepts are being utilized, and then look for divergences between the academic theory versus code implmentations to find issues
<johnny9dev>
it certainly sounds like that will be valuable as a shared harness project
<Rob1Ham>
I'm far from the final expert, so if nothing else can be a handy resource others can use to build even better security tools
nervana21 has quit [Ping timeout: 246 seconds]
jerryf_ has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
pseudoramdom has quit [Remote host closed the connection]
twistedline has quit [Ping timeout: 252 seconds]
<johnny9dev>
Rob1Ham: thank you for putting effort towards this
jerryf_ has quit [Remote host closed the connection]
luke-jr_ has joined #bitcoin-core-dev
jerryf has joined #bitcoin-core-dev
tigerMafia has quit [Quit: tigerMafia]
<Rob1Ham>
johnny9dev: happy to help, while I havent opened the code yet, i'm an open book about what I'm doing and happy to help anyone however I can as they explore adding the tools
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
bugs_ has quit [Quit: Leaving]
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
pseudoramdom has joined #bitcoin-core-dev
jeanpablojp has joined #bitcoin-core-dev
jerryf_ has joined #bitcoin-core-dev
jerryf has quit [Ping timeout: 261 seconds]
pseudoramdom has quit [Remote host closed the connection]
pseudoramdom has joined #bitcoin-core-dev
pseudoramdom has quit [Remote host closed the connection]
jerryf_ has quit [Remote host closed the connection]
___nick___ has quit [Ping timeout: 262 seconds]
jerryf has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 248 seconds]
dzxzg has joined #bitcoin-core-dev
jonatack has joined #bitcoin-core-dev
pseudoramdom has joined #bitcoin-core-dev
jerryf_ has joined #bitcoin-core-dev
cncr04s_ has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
amiti_____ has joined #bitcoin-core-dev
pseudoramdom has quit [Remote host closed the connection]
hebasto_ has joined #bitcoin-core-dev
corebot` has joined #bitcoin-core-dev
cncr04s has quit [Ping timeout: 240 seconds]
corebot has quit [Ping timeout: 254 seconds]
S3RK has quit [Ping timeout: 264 seconds]
S3RK has joined #bitcoin-core-dev
hebasto has quit [Ping timeout: 259 seconds]
amiti____ has quit [Ping timeout: 259 seconds]
hebasto_ is now known as hebasto
<pinheadmz>
fanquake #35632 is in the 32 milestone but how urgent is that. I think the most proper fix is probably in #35780 but that wont be ready for a while. is it worth opening a short term fix for the intermittent test before branch off?
<corebot`>
https://github.com/bitcoin/bitcoin/issues/35632 | intermittent ci failure: interface_http.py", line 558, in check_whitespace_in_headers // ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine · Issue #35632 · bitcoin/bitcoin · GitHub
<corebot`>
https://github.com/bitcoin/bitcoin/issues/35780 | http: linger-close after parse errors so clients can read the reply by b-l-u-e · Pull Request #35780 · bitcoin/bitcoin · GitHub
jerryf_ has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
corebot` is now known as corebot
jerryf_ has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf_ has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
l0rinc has quit [Quit: l0rinc]
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
hacker4web3bitco has quit [Ping timeout: 261 seconds]
hacker4web3bitco has joined #bitcoin-core-dev
corpus has joined #bitcoin-core-dev
Rob1Ham has quit [Ping timeout: 257 seconds]
corpus has quit [Client Quit]
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jonatack has quit [Ping timeout: 255 seconds]
jonatack has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf_ has joined #bitcoin-core-dev
jerryf_ has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf_ has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
<fanquake>
pinheadz: if you're get a short-term fix in mind, worthwhile I think
<fanquake>
(at least for a look)
twistedline has joined #bitcoin-core-dev
hacker4web3bitco has quit [Ping timeout: 241 seconds]
twistedline has quit [Remote host closed the connection]
Rob1Ham has quit [Ping timeout: 253 seconds]
twistedline has joined #bitcoin-core-dev
jadi has joined #bitcoin-core-dev
twistedline has quit [Ping timeout: 243 seconds]
l0rinc has joined #bitcoin-core-dev
twistedline has joined #bitcoin-core-dev
dzxzgtwo has quit [Quit: onversation terminated!]
jerryf has joined #bitcoin-core-dev
twistedline has quit [Ping timeout: 273 seconds]
twistedline_ has joined #bitcoin-core-dev
jerryf_ has quit [Remote host closed the connection]
jerryf_ has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
twistedline has joined #bitcoin-core-dev
jerryf_ has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
twistedline_ has quit [Ping timeout: 261 seconds]
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jadi has quit [Quit: jadi]
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf_ has joined #bitcoin-core-dev
jerryf_ has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
tigerMafia has joined #bitcoin-core-dev
tigerMafia has quit [Client Quit]
winterrdog has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
<winterrdog>
pinheadmz: i had worked on sth similar in my repo before but never committed/pushed it since i found a similar PR open. i’ve committed what i had and pushed it here: https://github.com/winterrdog/bitcoin/pull/2. might be of help, take a look
smartin has quit [Remote host closed the connection]
jerryf_ has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
winterrdog has quit [Quit: winterrdog]
jerryf_ has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jonatack has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
Guest18 has quit [Quit: Client closed]
jerryf has quit [Remote host closed the connection]
jerryf_ has joined #bitcoin-core-dev
SpellChecker_ has quit [Quit: bye]
afiore has quit [Quit: GNU inetutils 1.3a (telnet)]
ghost43 has quit [Quit: Leaving]
SpellChecker has joined #bitcoin-core-dev
ghost43 has joined #bitcoin-core-dev
afiore has joined #bitcoin-core-dev
jerryf_ has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
memset has quit [Ping timeout: 264 seconds]
memset has joined #bitcoin-core-dev
memset has quit [Remote host closed the connection]
andrewtoth has quit [Remote host closed the connection]
andrewtoth_ has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
memset has joined #bitcoin-core-dev
_andrewtoth_ has joined #bitcoin-core-dev
andrewtoth_ has quit [Remote host closed the connection]
jerryf has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
jerryf has quit [Remote host closed the connection]