abubakarsadiq has quit [Quit: Connection closed for inactivity]
Guest82 has quit [Quit: Client closed]
Guesttytty2 has joined #bitcoin-core-dev
Guesttytty2 has quit [Changing host]
Guesttytty2 has joined #bitcoin-core-dev
<Guesttytty2>
hello
<Guesttytty2>
iam trying to link my local fuzzer to bitcoin node(process_message target in this case). first I run
<Guesttytty2>
cmake -B build \ 127 [13:15:33] │
<Guesttytty2>
│ -DBUILD_FOR_FUZZING=ON \ │
<Guesttytty2>
│ -DCMAKE_C_COMPILER=clang \ │
<Guesttytty2>
│ -DCMAKE_CXX_COMPILER=clang++ \ │
<Guesttytty2>
│ -DFUZZ_LIBS="/path/to/.a files"
<Guesttytty2>
and then
<Guesttytty2>
cmake --build build
<Guesttytty2>
both commands work fine
<Guesttytty2>
then when I try to run FUZZ=process_message ./build/bin/fuzz ./corpus/process_message
<Guesttytty2>
I get FUZZ target 'process_message' not found. I wonder if there is an additional step I am missing to let my local fuzzer know about bitcoin targets?
<fanquake>
You should use FUZZ=process_message ./build/bin/fuzz
<fanquake>
Ah, sorry, misread the corpus dir
<dergoegge>
You can try with PRINT_ALL_FUZZ_TARGETS_AND_ABORT=1 to see all harnesses that are compiled in
<Guesttytty2>
Yea my problem is exactly that, I dont see any harness compiled when I do
<dergoegge>
If you're building your own fuzzer you'll need to make sure it also calls "LLVMFuzzerInitialize"
<fanquake>
Please stop pasting over multiple lines
<dergoegge>
For sharing terminal output I would recommend something like https://pastebin.com/ :)
<Guesttytty2>
oops sorry. Ok Ill have a look at LLVMFuzzerInitialize. Thank you.
kevkevin has quit [Ping timeout: 256 seconds]
Guest98 has joined #bitcoin-core-dev
Guest98 has quit [Client Quit]
<Guesttytty2>
Interestingly `PRINT_ALL_FUZZ_TARGETS_AND_ABORT=1 ./build/bin/fuzz ` does return now(after adding LLVMFuzzerInitialize) a list that contain 'process_message' and others. But I still see the not found error from above.
kevkevin has joined #bitcoin-core-dev
<Guesttytty2>
If I try to fuzz `wallet_*` I see `No fuzz target compiled for wallet_*..`. Everything else return `Error: Fuzz target '*' not found`.
kevkevin has quit [Ping timeout: 260 seconds]
<maflcko>
Guesttytty2: You'll have to compile with the wallet. I'd suggest to share exact and full steps to reproduce (including the configure summary, etc)
Guyver2 has quit [Remote host closed the connection]
<Guesttytty2>
Third command is `FUZZ=process_message ./build/bin/fuzz ./corpus/process_message` and it return `Error: FUZZ target 'process_message' not found`
jerryf_ has quit [Remote host closed the connection]
jerryf has joined #bitcoin-core-dev
<dergoegge>
"Error: FUZZ target 'process_message' not found" does not look like an error message that Bitcoin Core produces. You should check if your fuzzer also expects a FUZZ env variable and fails to interpret the "process_message" value.
<dergoegge>
This discussion might be too noisy for this channel, you can DM me here on irc, I'll try to give you some pointers
<maflcko>
Guesttytty2: What is git log -1 and git status?
<b10c>
gmaxwell: re 82.66.103.79: it's an altcoin node with Bitcoin mainnet network magic
<Guesttytty2>
maflcko I am at f490f5562d4b20857ef8d042c050763795fd43da (29 tag, without changes). Ill move the discussion to a private chat with dergoegge.
<l0rinc>
fanquake: we don't know the future, no need to state it that aggressively - we should discuss dropping it separately, especially since the release notes already does that
<dergoegge>
what's the point of keeping those options around?
<achow101>
to reduce drama lol
<jon_atack>
fanquake: it is a longstanding bug (perhaps first reported in 2014 (https://github.com/bitcoin/bitcoin/issues/5097), for me it made a very large difference in IBD time, though if others disagree I suppose it can wait for another release
<l0rinc>
yes
<l0rinc>
and to unify the wording with the release notes
<dergoegge>
yea lol
<achow101>
jon_atack: long standing bugs generally don't meet the bar for merging after feature freeze
<fanquake>
jonatack: probably good to add a test/reproducer then?
<dergoegge>
I don't think we should keep code around to "reduce drama" but I also don't care that much
MrHAPPY has joined #bitcoin-core-dev
<janb84>
at the time, the code is still there and the project has some history to un-deprecate options. so why the premature harshness
<l0rinc>
exactly - we're not proposing to keep it, just to discuss the removal separately
<dergoegge>
just seems more honest to say it will be removed if that's the plan (i thought it was)
<achow101>
hmm, we're past translation strings freeze too, so changing this will result in yet another translations update
<l0rinc>
no, it's more honest to say that "the plan is to remove"
<fanquake>
Yea. strings have already been updated and frozen
<l0rinc>
I can update the translations as well in the PR if needed
<achow101>
there isn't really a semantics change, so I don't think it should be in the milestone
<jon_atack>
regarding the outstanding question raised in 32051, dropping the middle two commits for now to only give addnode peers more time would resolve it
<jon_atack>
I'll look into doing that
<dzxzg>
I think the string is fine as-is, the ambiguity is embedded in the fact that it's the PoV of the project at time of release, and obviously that perspective can change in the future, I don't think much is gained by explicitly laying out that the future is hard to predict.
<achow101>
Anything else to discuss?
<stickies-v>
i don't have a strong opinion on the string change, either is fine by me, but it doesn't seem worth doing another translations update to me
<l0rinc>
I don't mind doing the extra work, but I think the reword could avoid extra drama (this line was pointed out on Twitter explicitly)
<achow101>
Should #32592 need to be in the milestone? It's needed rebase for 2 months
<fanquake>
I don't see why we'd change it under the premise of having the discussion again, and maybe not dropping it, when that has already been decided, adn rejected in #32714
<corebot>
https://github.com/bitcoin/bitcoin/issues/32714 | init, doc: Replace datacarrier(size) deprecation with non-recommendation text by achow101 · Pull Request #32714 · bitcoin/bitcoin · GitHub
<TheCharlatan>
I'd want #32592, would be nice to not have to backport to the old macros.
<jon_atack>
"long standing bugs generally don't meet the bar for merging after feature freeze" -> hm, I understand, nevertheless, only recent bugs meeting the bar seems a little odd, idk
Cory23 has joined #bitcoin-core-dev
Cory66 has joined #bitcoin-core-dev
<jon_atack>
i don't think unifying the messages is bikeshedding, nor that unifying them would add drama (quite the contrary)
<janb84>
jon_atack: if they are there for quite some time, why the rush to include them after a freeze. There was no need before. And newer bugs could potentially be higher prio ;) that's the thinking i guess
Cory2 has quit [Ping timeout: 250 seconds]
Cory23 has quit [Ping timeout: 250 seconds]
<jon_atack>
janb84: generally a PR shouldn't be rush-merged, it's only a question whether the fix is desired if well-reviewed and ready. A PR can always be dropped from the milestone if not.
Cory12 has joined #bitcoin-core-dev
Cory66 has quit [Ping timeout: 250 seconds]
<janb84>
jon_atack: true true
<instagibbs>
Bugs which aren't new aren't blockers for people updating, new bugs are
w0xlt has joined #bitcoin-core-dev
<jon_atack>
instagibbs: Looking at the 30.0 milestone, it contains some items that aren't bug fixes. Some are only fixups or nice-to-have cleanups. I think it is fine to include them, just odd to at the same time exclude a bug fix that can affect IBD on a slow connection.
<jon_atack>
Anyway, all I can do is propose and move on. Cheers
durandal_ has joined #bitcoin-core-dev
bitdex has joined #bitcoin-core-dev
_durandal has quit [Ping timeout: 248 seconds]
<lightlike>
jon_atack: would be good to address feedback, it's not been updated for months - it's not like this is a simple bugfix with no possible downsides. e.g. I don't really understand _why_ the suposedly high-quality addnode peers are being disconnected.
<lightlike>
were they much slower to deliver blocks compared to other peers? If so, doesn't giving them more time before disconnection make IBD even slower (if there are faster peers available)?
<lightlike>
If they weren't slower compared to other peers, then why were they marked as the staller? Maybe by mistake, due to the scenario described in #32179? Or some other reason?
Novo__ has quit [Quit: Connection closed for inactivity]
Cory80 has joined #bitcoin-core-dev
Cory12 has quit [Ping timeout: 250 seconds]
Ademan has joined #bitcoin-core-dev
<jon_atack>
lightlike: they were being disconnected like automated peers due to a slow connection during ibd, then reconnected immediately, 100+ times per minute, because the thread reconnects addnode peers immediately. when i then ran bitcoind with that pull for ibd, the issue was resolved.
<jon_atack>
I think I described this well in the PR description, but perhaps I'm not understanding your questions. Will look into it more (thanks!)
<jon_atack>
(I assume by default that I'm not understanding fully what you are writing, so will investigate with that in mind.)
<jon_atack>
This was happening not because the peers were low quality, but because my connection was slower than that allowed by our stalling logic.
<jon_atack>
s/100+ times per minute/100+ times per hour/
<lightlike>
"the issue was resolved" - ok, but was IBD faster? after all that is the main goal, not minimizing the number of connections we go make.
<jon_atack>
The simplest fix would be the first commit, giving addnode peers more time before considered stalling, while dropping the next two commits that exempt them. The first commit is the one that fixed my IBD issue.
mudsip has joined #bitcoin-core-dev
mudsip has quit [Client Quit]
<lightlike>
wouldn't the first commit also slow down IBD in case the addnode peer is slower compared to other peers?
<jon_atack>
I think that potential tradeoff is worth it. IBD was faster in that it became somewhat more possible, from 6-7 weeks down to 2-3 weeks. Though the connection speed could have been different too. The debug log was longer full of disconnections and reconnections. I have addnode peers that are clearnet, cjdns (as fast as clearnet), i2p (resource-intensive to build two tunnels for each
<jon_atack>
connection), and tor.
<jon_atack>
Been thinking if we ought to describe in the docs what kind of peers to select for addnode, e.g. *trusted* ones that the user knows personally.
MyNickname has quit [Read error: Connection reset by peer]
<l0rinc>
Could you add your IBD measurements and maybe some plot that shows the disconnects before/after based on the debug.log?
<jon_atack>
l0rinc: sgtm
* jon_atack
lunch
l0rinc has quit [Read error: Connection reset by peer]
<lightlike>
i think that tor addnode peers during IBD, if there are also clearnet peers, are usually a bad idea - they are much slower compared to other peers, the stalling mechanism tries to kick them, when it does we reconnect, so they just slow us you down without any benefit to us or them (that changes obviously once synced).
<lightlike>
i guess this doesn't really apply if our connection is even slower than the average tor peer's speed.
VonNaturAustreVe has quit [Ping timeout: 248 seconds]
<Sjors[m]1>
FYI I plan to go through all my non-IPC pull requests next week and deal with feedback (mainly wallet stuff).
cfields has joined #bitcoin-core-dev
l0rinc has joined #bitcoin-core-dev
VonNaturAustreVe has joined #bitcoin-core-dev
Guest15 has joined #bitcoin-core-dev
entropyx has quit [Ping timeout: 244 seconds]
Guest15 has quit [Client Quit]
Guest15 has joined #bitcoin-core-dev
l0rinc has quit [Ping timeout: 248 seconds]
entropyx has joined #bitcoin-core-dev
Guest15 has quit [Client Quit]
Guesttytty2 has quit [Quit: Client closed]
eugenesiegel64 has quit [Ping timeout: 250 seconds]
iSiUp has quit [Quit: WeeChat 4.7.1]
bitcoinlover has quit [Ping timeout: 272 seconds]
l0rinc has joined #bitcoin-core-dev
iSiUp has joined #bitcoin-core-dev
jonatack has joined #bitcoin-core-dev
jon_atack has quit [Ping timeout: 245 seconds]
l0rinc has quit [Ping timeout: 258 seconds]
iSiUp has quit [Quit: WeeChat 4.7.1]
<gmaxwell>
the whole use of stalling disconnection as a performance optimizer is not really an intended or designed feature. It's not what you'd come up with if you went in and said "I want to make IBD faster", but rather it's just a product of the initial out of order fetching code needing some escape hatch so that a single broken peer wouldn't completely jam progress forever.
_durandal has joined #bitcoin-core-dev
<gmaxwell>
And the way all this logic is contructed tends to produce bad tradeoffs. Like if you set the timeouts short, then nodes with slow connections are at risk of making no progress or only making progress slowly and inefficiently due to congestion collapse.
<gmaxwell>
(where they terminate peers with inflight blocks, which are consuming resources in the background, just to replace with someone else who also will get terminated in flight).
durandal_ has quit [Ping timeout: 248 seconds]
enochazariah has joined #bitcoin-core-dev
l0rinc has joined #bitcoin-core-dev
probot has joined #bitcoin-core-dev
iSiUp has joined #bitcoin-core-dev
Talkless has quit [Quit: Konversation terminated!]
Randolf has joined #bitcoin-core-dev
Randolf has quit [Client Quit]
<lightlike>
it has that effect though, that non-broken peers that are much slower compared to others are removed over time, though in a pretty non-aggressive way (compared to libbitcoin for example). I always thought this was some kind of compromise between IBD speed and distributing IBD load over many peers, not just the fastest ones.
Cory2 has joined #bitcoin-core-dev
Cory80 has quit [Ping timeout: 250 seconds]
l0rinc has quit [Ping timeout: 248 seconds]
enochazariah has quit [Ping timeout: 250 seconds]
<gmaxwell>
yes it does, but it's just incidental. We knew that it would improve performance and considered it fortunate. But it's not a *good* way to so, particularly since tweaking down the timeouts to try to make it more effective will kill slower (or DOS attacked) peers from syncing at all.
<gmaxwell>
it was never intended as any kind of compromise between speed and distributing over many peers.
<gmaxwell>
and it was developed under pressure because the original syncing mechenism was beginning to fail to work completely.