<bitcoin-git>
bitcoin/master 31fdc54 kdmukai: test: speed up wallet_fundrawtransaction.py and wallet_sendall.py
<bitcoin-git>
bitcoin/master 0139a0d MarcoFalke: Merge bitcoin/bitcoin#26722: test: speed up the two slowest functional tes...
<bitcoin-git>
[bitcoin] MarcoFalke merged pull request #26722: test: speed up the two slowest functional tests by 18-35% via `keypoolrefill()` (master...wallet_fundrawtransaction_speedup) https://github.com/bitcoin/bitcoin/pull/26722
sipsorcery has joined #bitcoin-core-dev
<bitcoin-git>
[gui] MarcoFalke merged pull request #687: Load PSBTs using istreambuf_iterator rather than istream_iterator (master...load-psbt-istreambuf) https://github.com/bitcoin-core/gui/pull/687
<bitcoin-git>
bitcoin/master bb5ea1d Andrew Chow: qt: Load PSBTs using istreambuf_iterator rather than istream_iterator
<bitcoin-git>
bitcoin/master 4cd6b3b MarcoFalke: Merge bitcoin-core/gui#687: Load PSBTs using istreambuf_iterator rather th...
<bitcoin-git>
[bitcoin] MarcoFalke opened pull request #26735: [24.x] qt: Load PSBTs using istreambuf_iterator rather than istream_iterator (24.x...2212-backport-gui-psbt-whitespace-fix-🔈) https://github.com/bitcoin/bitcoin/pull/26735
MrFrancis has quit [Remote host closed the connection]
MrFrancis has joined #bitcoin-core-dev
josie has quit [Ping timeout: 246 seconds]
ishaanam[m] has quit [Ping timeout: 246 seconds]
stratospher[m] has quit [Ping timeout: 246 seconds]
laanwj has quit [Ping timeout: 246 seconds]
jakwolf[m] has quit [Ping timeout: 265 seconds]
denise[m] has quit [Ping timeout: 264 seconds]
sipa has quit [Ping timeout: 246 seconds]
ShohamChakrabort has quit [Ping timeout: 252 seconds]
Murch1 has quit [Ping timeout: 260 seconds]
Jiri[m] has quit [Ping timeout: 248 seconds]
vincenzopalazzo has quit [Ping timeout: 252 seconds]
dunxen has quit [Ping timeout: 265 seconds]
willcl_ark has quit [Ping timeout: 265 seconds]
bitcoin-git has quit [Ping timeout: 260 seconds]
virtu has quit [Ping timeout: 260 seconds]
m_p78 has joined #bitcoin-core-dev
BlueMatt[m] has quit [Ping timeout: 260 seconds]
provoostenator has quit [Ping timeout: 246 seconds]
kakolainen[m]1 has quit [Ping timeout: 264 seconds]
kakolainen[m]1 has joined #bitcoin-core-dev
josie has joined #bitcoin-core-dev
ishaanam[m] has joined #bitcoin-core-dev
jakwolf[m] has joined #bitcoin-core-dev
stratospher[m] has joined #bitcoin-core-dev
denise[m] has joined #bitcoin-core-dev
Jiri[m] has joined #bitcoin-core-dev
willcl_ark has joined #bitcoin-core-dev
ShohamChakrabort has joined #bitcoin-core-dev
Murch1 has joined #bitcoin-core-dev
Guyver2 has joined #bitcoin-core-dev
BlueMatt[m] has joined #bitcoin-core-dev
vincenzopalazzo has joined #bitcoin-core-dev
dunxen has joined #bitcoin-core-dev
sipa has joined #bitcoin-core-dev
bitcoin-git has joined #bitcoin-core-dev
virtu has joined #bitcoin-core-dev
provoostenator has joined #bitcoin-core-dev
laanwj has joined #bitcoin-core-dev
Guyver2 has quit [Remote host closed the connection]
m_p78 has left #bitcoin-core-dev [#bitcoin-core-dev]
brunoerg has joined #bitcoin-core-dev
AaronvanW has quit [Remote host closed the connection]
AaronvanW has joined #bitcoin-core-dev
virtu has quit [Changing host]
virtu has joined #bitcoin-core-dev
SpellChecker has quit [Quit: bye]
SpellChecker has joined #bitcoin-core-dev
ghost43_ has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] brunoerg opened pull request #26738: test: add coverage for unknown wallet flag in `setwalletflag` (master...2022-12-setwalletflag-unknown-error) https://github.com/bitcoin/bitcoin/pull/26738
ghost43 has quit [Ping timeout: 255 seconds]
AaronvanW has quit [Ping timeout: 260 seconds]
brunoerg has quit []
Cory has quit [Ping timeout: 268 seconds]
AaronvanW has joined #bitcoin-core-dev
MacroFake has quit [Quit: ZNC 1.7.5+deb4 - https://znc.in]
MacroFake has joined #bitcoin-core-dev
AaronvanW has quit [Remote host closed the connection]
Guyver2 has joined #bitcoin-core-dev
MrFrancis has quit [Remote host closed the connection]
MrFrancis has joined #bitcoin-core-dev
Guyver2 has left #bitcoin-core-dev [Closing Window]
AaronvanW has joined #bitcoin-core-dev
MrFrancis has quit [Ping timeout: 252 seconds]
taylor has joined #bitcoin-core-dev
taylor is now known as Guest4411
yashraj has joined #bitcoin-core-dev
Guest4411 has quit [Quit: Client closed]
taylorfrommynode has joined #bitcoin-core-dev
AaronvanW has quit [Ping timeout: 248 seconds]
yashraj_ has joined #bitcoin-core-dev
yashraj has quit [Ping timeout: 246 seconds]
hg has joined #bitcoin-core-dev
hernanmarino has joined #bitcoin-core-dev
yashraj_ has quit [Ping timeout: 260 seconds]
kmartin has joined #bitcoin-core-dev
pablomartin has joined #bitcoin-core-dev
rozehnal_paul has joined #bitcoin-core-dev
b_101_ has joined #bitcoin-core-dev
b_101 has quit [Ping timeout: 272 seconds]
kmartin has quit [Quit: Client closed]
as2333 has joined #bitcoin-core-dev
p2plife has quit [Quit: quit]
b_101 has joined #bitcoin-core-dev
b_101_ has quit [Read error: Connection reset by peer]
<bitcoin-git>
bitcoin/master f3bc1a7 Andrew Chow: Merge bitcoin/bitcoin#26265: POLICY: Relax MIN_STANDARD_TX_NONWITNESS_SIZE...
sipsorcery has joined #bitcoin-core-dev
rozehnal_paul has quit [Quit: Connection closed]
hernanmarino has quit [Quit: Leaving]
b_101 has joined #bitcoin-core-dev
b_101_ has quit [Ping timeout: 246 seconds]
<p2plife>
user on #bitcoin claims after upgrade to 0.24 some old wallets broke. taylorfrommynode (voice in #bitcoin)
<p2plife>
mentioning here since sounds like the kind of thing someone really should investigate
<achow101>
p2plife: thanks, looking
andrewtoth has joined #bitcoin-core-dev
<taylorfrommynode>
OK, I can post here now as well. I'm seeing a pretty prevalent issue when upgrading devices from v23.0 to v24.0 where Bitcoin fails to start, often on the second restart after upgrading to v24. The log reports a message, but it isn't overly helpful and just says wallet.dat cannot be opened.
<achow101>
taylorfrommynode: How have you specifieid the wallet to be opened?
<taylorfrommynode>
In bitcoin.conf like this
<taylorfrommynode>
main.wallet=wallet.dat
<achow101>
can you try `main.wallet=`
<achow101>
you're not actually supposed to give the relative path to the wallet.dat file
<taylorfrommynode>
wallet.dat: SQLite 3.x database, application id 4190024921, last written using SQLite version 3038005
<achow101>
can you double check the permissions on the file? make sure that the user that bitcoind will run as can read and write the file
<instagibbs>
does downgrading fix it? I'd check that as well
<achow101>
I'm fairly certain that nothing in the database opening code changed between 23.0 and 24.0
<taylorfrommynode>
Unfortunately, downgrading does not fix it. Some users report that is works the first time they launch 24 and it breaks after the first restart on 24.
<taylorfrommynode>
I tried tinkering with permissions and didn't see a difference.
<taylorfrommynode>
Released, downloaded from bitcoincore.org
<taylorfrommynode>
Also, I have reports of there being issues on PCs and Raspis so it doesn't seem to be arch-specific
<p2plife>
probably not but, taylorfrommynode, by any chance anything in dmesg logs, or any special protections on the system such as file ACLs there, apparmor and alike, selinux, being remote or otherwise special file system?
<instagibbs>
no expert but from reading the thing that's failing a couple quick checks: "head -c 16 <wallet file>" should result in "SQLite format 3", and check the file size of the wallet to see if it's weirdly small(<512 bytes)
<sipa>
Could there be two instances of bitcoind? With different datadir, but the same wallet?
<instagibbs>
inability to open seems most likely...
<sipa>
Is bitcoind running as user bitcoin?
<sipa>
Is it a network drive?
<taylorfrommynode>
I have not seen any messages about security protections causing issues. The head command does return "SQLite format 3" and the size seems normal.
<achow101>
The only thing that makes sense for hitting this error is just simply failing to open the file
<instagibbs>
yeah only other thing would be the file header/magic being clobbered, which it doesn't look like
<sipa>
achow101: Is the sqlite version the same between 23.0 and 24.0.1?
<achow101>
file is detecting the magic and app id correctly, so it's not that
<p2plife>
can you open that file in sqlite program such as sqlite-browser (or how ever it was called)? does it seem fine there? BTW. ofc do backups of everything before all this
<instagibbs>
achow101, ah nice, TIL
<instagibbs>
so yeah, file permissions something something
<achow101>
sipa: I think it did, but not meaningfully
<taylorfrommynode>
I don't think there are two instances of bitcoind running since it loads fine if I don't load wallet.dat at startup or if disablewallet=1 is set.
<achow101>
taylorfrommynode: you are able to load the wallet with loadwallet?
<taylorfrommynode>
I'm trying loadwallet now to see if I get more info
Talkless has quit [Quit: Konversation terminated!]
<taylorfrommynode>
What seems extra concerning is that some users report this error even after removing the bad wallet or removing it from bitcoin.conf, they still get an error about the format of the whole datadir.
<taylorfrommynode>
Error: Failed to load database path '/home/bitcoin/.bitcoin'. Data is not in recognized format.
<achow101>
Can you start without a wallet, then make one with createwallet, and then unload and reload it?
<achow101>
is the problem with just this one specific wallet, or in general?
Guest16 has joined #bitcoin-core-dev
<sipa>
Can you try cat /path/to/wallet.dat | cut -c 69-72 | hd ?
<p2plife>
strace - maybe run in strace to see which exact files it tries to load from disk, |grep open. also ***DO NOT PASTE*** strace as it might show part of the secret data, in form of read(....) = "..." (parts of actuall WALLET secrets)
<instagibbs>
can you double check permissions of the containing directory?
Guest16 has quit [Client Quit]
<sipa>
taylorfrommynode: Sorry, -b instead of -c
<taylorfrommynode>
instagibbs: Folder perms seem fine, it runs without this wallet.
<taylorfrommynode>
sipa: -b doesn't seem to change the output
<instagibbs>
I see the magic, it's there
<sipa>
oh yes, nvm
<sipa>
all good
<achow101>
It has to be the `if (!file.is_open()) return false;` line
<achow101>
just no idea why the file isn't open
<willcl_ark>
Have we tried restarting the machine? :P
<p2plife>
taylorfrommynode, are all problems on same machine? what system(s), cpu architecture(s)?
<sipa>
taylorfrommynode: Do you have the same problems with self-compiled binaries?
pablomartin_ has quit [Ping timeout: 268 seconds]
<taylorfrommynode>
willcl_ark: Yeah, and downgrading back to v23.0.
<taylorfrommynode>
p2plife: A large number of devices are seeing this issue, mini pcs and raspberry pis.
<achow101>
hmm, presumably the datadir is on an external drive? Could it be a usb issue?
<taylorfrommynode>
sipa: I can't reproduce the initial wallet corruption, I've only got access to already corrupted files so I can't say right now.
<achow101>
taylorfrommynode: can you share the problematic wallet with us?
<taylorfrommynode>
achow101: The first issues were all reported on PCs using internal SATA drives
* p2plife
volounteers ;)
<p2plife>
right, are any of these wallets empty or almost empty?
MrFrancis has quit [Ping timeout: 246 seconds]
<taylorfrommynode>
p2plife: The couple I have seen all seem non-empty. I went through the same process with an empty wallet and saw no issue.
<p2plife>
taylorfrommynode, do the wallets load normally in sqlite-browser? should be fairly easy for remote user/owner to check that
<p2plife>
just let the user test first can he even read the file (e.g. calculate sha1sum of the same file as he is about to try opening) as he will probably do it from another user (needs GUI)
<taylorfrommynode>
What would look "normal"? I just opened one I was sent in sqlite browser.
<achow101>
mainly looking to see if sqlite complains about it
<taylorfrommynode>
I see a main table with key/value pairs and 5 keys. No obvious complaints.
<achow101>
can you try adding group read-write permissions to the wallet?
<achow101>
e.g. chmod g+rw wallet.dat
AaronvanW has joined #bitcoin-core-dev
<taylorfrommynode>
Changed permissions, no difference.
<sipa>
Do self-compiled binaries also error out on this corrupted wallet.dat file?
<taylorfrommynode>
I see 5 keys - ".version" "..." "<weird character>flags" " bestblock" "<arrow char>bestblock_nomerkle"
<achow101>
that sounds about right for a watchonly
<taylorfrommynode>
Why would the keys have such crazy characters though? Makes me think it's corrupt somehow.
<p2plife>
are these watchonly wallets, not with actually private keys? then they perhaps might be shared (minus the privacy consideration of linking all addresses as same user)
<achow101>
the keys contain binary data
<achow101>
with strings sometimes
<p2plife>
what table should he see with the 1000+ privkeys for normal wallet?
MrFrancis has joined #bitcoin-core-dev
<achow101>
something like 30 ish records, with several that have a string that start with "walletdescriptor"
<p2plife>
is there no exact debug logging steps of trying to load a wallet, to be turned on? if not then maybe my above strace idea (just do not paste content of read file data)
<taylorfrommynode>
I turned on all debug and it did not add any additional logs unfortunately
<achow101>
there isn't. strace is probably a good idea
<sipa>
(as something to add to make it report why it fails)
<sipa>
s/files/fails/
<p2plife>
is there some walletinfo you can paste? maybe with exact wallet type (descriptors? HD? how many keys? watchonly? privkeys? ... more?)
<taylorfrommynode>
As far as I know it's watch only. Some apps require or used to use it at least, but none store funds there (unless someone manually tried to but I seriously doubt that).
<p2plife>
so... it gets angry it can't open wallet.dat as a directory after just opening it as a file??
<achow101>
that's a funny error code
<achow101>
I think I know what went wrong
<achow101>
taylorfrommynode: can you create a new directory inside the datadir and put the wallet.dat in there. Then in the config (or loadwallet), instead of specifying wallet.dat, specify the name of the directory
<p2plife>
I would report this bug on github, sadly github shadowbans new users from emails it doesn't like. Would be great to one day move off that central platform
<p2plife>
besides this work-around, this will be patched up then?
<taylorfrommynode>
Success! Putting it inside a folder causes bitcoin to start as expected.
<p2plife>
excellent success and as usual top notch developer's support for Bitcoin
<achow101>
Here's what I think is happening. Some third party software is making the wallets and also manually manipulating the location of the wallet files on disk. Said software was using legacy wallets, which allow for plain wallet files in the walletsdir (which may also be the datadir). With 24.0, those software are also calling migratewallet to turn those legacy wallets into descriptor wallets. However we do not support wallet files directly in the
<achow101>
walletsdir for descriptor wallets and always assume the given wallet name is the directory containing the wallet.dat. Hence we get the error that open() tries to open the file as a directory.
orville has joined #bitcoin-core-dev
<achow101>
It's possible that the migration tool is accidentally making the wallet files in the wrong location, but I don't think that's the case since we use the wallet creation utilities which will always make new directories for the wallets.
<achow101>
I'll need to double check that though
<achow101>
the altnerative is that the third party software is copying the wallet files around and expecting the sqlite wallets to have the same behavior as bdb, which they do not.
<p2plife>
is all of this in some README? types of wallet formats, which location can be used for which (and since what version it changed)? sounds important to have
<sipa>
achow101: Is there a reason for not supporting descriptor wallets in arbitrary locations?
<achow101>
no, but users manipulating the file locations isn't something that's really supported
<achow101>
sipa: I vaguely recall a discussion about this in the pr that added sqlite, but not quite what the reasoning was, other than that this was consistent with how we wanted wallets to be handled, and since descriptor wallets were completely new, we weren't expecting backwards compatibility issues
<sipa>
Makes sense.
grubmn has quit [Quit: WeeChat 1.9.1]
yanmaani1 has quit [Remote host closed the connection]
yanmaani1 has joined #bitcoin-core-dev
andrewtoth has quit [Remote host closed the connection]
MrFrancis has quit [Ping timeout: 248 seconds]
andrewtoth has joined #bitcoin-core-dev
as2333 has quit [Quit: as2333]
as2333 has joined #bitcoin-core-dev
andrewtoth has quit [Remote host closed the connection]
andrewtoth has joined #bitcoin-core-dev
<achow101>
apparently migrating plain wallet files just doesn't work, so I guess it's file manipulation that's the problem
taylorfrommynode has quit [Quit: Client closed]
taylorfrommynode has joined #bitcoin-core-dev
andrewtoth has quit [Remote host closed the connection]
andrewtoth has joined #bitcoin-core-dev
kexkey has quit [Read error: Connection reset by peer]
kexkey has joined #bitcoin-core-dev
kexkey has quit [Client Quit]
andrewtoth has quit [Quit: Leaving]
kexkey has joined #bitcoin-core-dev
andrewtoth has joined #bitcoin-core-dev
<bitcoin-git>
[bitcoin] achow101 opened pull request #26740: wallet: Migrate wallets that are not in a wallet dir (master...migrate-plain-wallet-file) https://github.com/bitcoin/bitcoin/pull/26740
kexkey has quit [Quit: kexkey]
kexkey has joined #bitcoin-core-dev
AaronvanW has joined #bitcoin-core-dev
hg has quit [Quit: WeeChat 3.7.1]
kexkey has quit [Quit: kexkey]
orville has quit [Quit: <exit all I/O streams / EOF>]
MrFrancis has joined #bitcoin-core-dev
TheRec_ has quit [Read error: Connection reset by peer]