<Krellan>
I wrote that, then ran into trouble trying to reproduce the original runaway exception that earlier intrigued me! seems the runaway exception only happens on the shipped 29.0 and not on what I compiled from the 29.0 branch. I'm guessing the version of Berkeley DB on my system is newer enough the wallet corruption/incompatibility didn't happen
kevkevin has quit [Ping timeout: 260 seconds]
entropyx has quit [Remote host closed the connection]
purpleKarrot has quit [Quit: purpleKarrot]
entropyx has joined #bitcoin-core-dev
entropyx has joined #bitcoin-core-dev
kevkevin has joined #bitcoin-core-dev
kevkevin has quit [Ping timeout: 272 seconds]
Guyver2 has quit [Quit: Going offline, see ya! (www.adiirc.com)]
charlie_capt has quit [Quit: Client closed]
kevkevin has joined #bitcoin-core-dev
jespada has joined #bitcoin-core-dev
Guyver2 has joined #bitcoin-core-dev
Guyver2 has left #bitcoin-core-dev [#bitcoin-core-dev]
kevkevin has quit [Ping timeout: 252 seconds]
kevkevin has joined #bitcoin-core-dev
kevkevin has quit [Ping timeout: 276 seconds]
jespada has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
jespada has joined #bitcoin-core-dev
kevkevin has joined #bitcoin-core-dev
pablomartin has quit [Quit: ZNC 1.8.2+deb3.1+deb12u1 - https://znc.in]
hernanmarino has quit [Quit: ZNC 1.8.2+deb2+deb11u1 - https://znc.in]
hernanmarino has joined #bitcoin-core-dev
kevkevin has joined #bitcoin-core-dev
Guest40 has joined #bitcoin-core-dev
Guest40 has quit [Client Quit]
kevkevin has quit [Ping timeout: 265 seconds]
Murch[m] has quit [Quit: Bridge terminating on SIGTERM]
bitcoin-git has quit [Quit: Bridge terminating on SIGTERM]
sr_gi[m] has quit [Quit: Bridge terminating on SIGTERM]
BlueMattMtrxBot has quit [Quit: Bridge terminating on SIGTERM]
laanwj has quit [Quit: Bridge terminating on SIGTERM]
Sjors[m] has quit [Quit: Bridge terminating on SIGTERM]
BlueMattMtrxBot has joined #bitcoin-core-dev
kvaciral[m] has joined #bitcoin-core-dev
kevkevin has joined #bitcoin-core-dev
BlueMattTest has joined #bitcoin-core-dev
BlueMatt[m] has joined #bitcoin-core-dev
spynx is now known as spynxic
stratospher[m] has joined #bitcoin-core-dev
sr_gi[m] has joined #bitcoin-core-dev
bitcoin-git has joined #bitcoin-core-dev
laanwj has joined #bitcoin-core-dev
Murch[m] has joined #bitcoin-core-dev
b10c[m] has joined #bitcoin-core-dev
Sjors[m] has joined #bitcoin-core-dev
jespada has quit [Ping timeout: 244 seconds]
jespada has joined #bitcoin-core-dev
robszarka has joined #bitcoin-core-dev
luke-jr_ has joined #bitcoin-core-dev
szarka has quit [Ping timeout: 248 seconds]
bugs_ has joined #bitcoin-core-dev
yuvic has joined #bitcoin-core-dev
yuvic has quit [Remote host closed the connection]
Talkless has joined #bitcoin-core-dev
bitdex has joined #bitcoin-core-dev
PaperSword has quit [Remote host closed the connection]
brunoerg_ has quit [Remote host closed the connection]
robobub has joined #bitcoin-core-dev
mudsip has joined #bitcoin-core-dev
mudsip has quit [Client Quit]
thoragh has joined #bitcoin-core-dev
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Ping timeout: 248 seconds]
Talkless has quit [Quit: Konversation terminated!]
bugs_ has quit [Quit: Leaving]
Guest79 has joined #bitcoin-core-dev
Guest79 has quit [Client Quit]
<bitcoin-git>
[bitcoin] achow101 opened pull request #32618: wallet: Remove ISMINE_WATCHONLY and watchonly from RPCs (master...delete-ismine-watchonly) https://github.com/bitcoin/bitcoin/pull/32618
brunoerg has joined #bitcoin-core-dev
robszarka has quit [Quit: Leaving]
szarka has joined #bitcoin-core-dev
Guest99 has joined #bitcoin-core-dev
<gmaxwell>
achow101: pardon my lack of following things but is there a util that will take a directory of legacy wallets and extra all their (potentially encrypted) keys, dedupe them, and dump all the keys into a new wallet?
<achow101>
gmaxwell: no
<gmaxwell>
the old disk scanning tool on bitcointalk does something like that but only writes out a legacy wallet and doesn't work on encrypted wallets.
<gmaxwell>
even without prior to the legacy removal scanning old wallets is kind of a nightmare as they have duplicate fileids so you can't load multiple at once plus the rescan runs once per wallet. I went through an old backup a while ago and it took about a month to scan all of them. oy.
<bitcoin-git>
[bitcoin] achow101 opened pull request #32619: wallet, rpc, gui: List legacy wallets with a message about migration (master...dont-list-legacy-wallets) https://github.com/bitcoin/bitcoin/pull/32619
<sipa>
gmaxwell: i guess you could convert them all to descriptor wallets, dump all the private descriptors, and import them all into a single wallet?
<sipa>
though you may end up with a gazillion descriptors, unsure how performance scales with those
<achow101>
sipa: I think the problem is all of the rescan on loading that will happen
<sipa>
ah right
<sipa>
well i guess you can do all of this with an empty datadir and -connect=0
<gmaxwell>
without encryption (and before the legacy wallet removal) a quick trick is to just cat all the wallets into a file then run https://bitcointalk.org/index.php?topic=25091.0 on it and the result is a single deduped legacy wallet.
<gmaxwell>
achow101: in any case thanks for all your recent wallet work!
brunoerg has quit [Remote host closed the connection]
javi404 has joined #bitcoin-core-dev
<phantomcircuit>
gmaxwell: iirc gavin actually wrote something that naively scans wallets looking for the bdb key format
<phantomcircuit>
for unencrypted wallets it worked but is slow because python
<phantomcircuit>
and like only maybe actually works
<gmaxwell>
the thing I linked on bitcointalk does that but only works on unencrypted wallets.
<gmaxwell>
and it absolutely works and is a total lifesaver.
<gmaxwell>
but it also only outputs a legacy wallet, which doesn't directly work with bitcoin core anymore.
<phantomcircuit>
gmaxwell: iirc bdb can actually end up writing things to disk across segments which breaks that kind of scanner
<gmaxwell>
phantomcircuit: probably an ideal tool would do both brute scraping and also trying to read with BDB.
<gmaxwell>
the scaping works on files that are so corrupt bdb won't touch them.
<phantomcircuit>
gmaxwell: yeah which is definitely better than nothin
<phantomcircuit>
g
<bitcoin-git>
[bitcoin] achow101 opened pull request #32620: wallet: Fix wallet interface detection of encrypted wallets (master...fix-gui-migrate-encrypted) https://github.com/bitcoin/bitcoin/pull/32620
SpellChecker_ has joined #bitcoin-core-dev
<gmaxwell>
I'm finding now that good brand sata/nvme SSDs that sat in a closet for 5 years are full of errors, so I assume this also means a lot of earlier bitcoin users might be finding their older wallets harder to recover.
ghost43_ has joined #bitcoin-core-dev
SpellChecker has quit [Ping timeout: 264 seconds]
<gmaxwell>
(mostly hasn't been a real problem for me, but old random dust wallets are now valuable...)
<phantomcircuit>
gmaxwell: flash rots overtime, turns out it's not possible to build a perfect capacitor
<phantomcircuit>
that they work at all is kinda insane really
<gmaxwell>
indeed, not news to me in theory, but interesting to see it in practice.
ghost43 has quit [Remote host closed the connection]
<phantomcircuit>
note that blurays will also do this
<bitcoin-git>
[gui] achow101 opened pull request #877: gui: Add a menu action to restore then migrate a legacy wallet (master...gui-migrate-path) https://github.com/bitcoin-core/gui/pull/877
<gmaxwell>
I wonder if powering up the SSD and doing a smart offline scan is sufficient to make the drive rewrite sectors with soft errors.
<phantomcircuit>
you'll lose entire sectors at random because writable discs made at home degrade from regular sunlight
<phantomcircuit>
gmaxwell: unlikely, you almost certainly have to do a full read write scan where you didn't write back identical blocks
<gmaxwell>
phantomcircuit: so I think not because I have some drives of the same make that have been ONLINE and they're fine, compared to ones offline that are full of errors, so I think the drive firmware does some maintance.
Christoph_ has joined #bitcoin-core-dev
<phantomcircuit>
gmaxwell: i suspect that when the drives are offline completely they just leak from the individual cells easier
<gmaxwell>
for electronic data backup I've used parallel DLT and archival CDR. though keeping DLT going is kind of a PITA.
<phantomcircuit>
the relative voltage potential is much higher
<phantomcircuit>
or i could be totally wrong on that, but it's the impression that i got
<gmaxwell>
I'm sure that some SSDs will rewrite soft errors, but you can never be sure about it because its all unspecified.
<phantomcircuit>
it's also going to be entirely drive dependent, cheap drives might not ever refresh cells, expensive ones might apply complex algorithsm that break, maybe someone makes a reasonable one that just background refreshes linearly, but i have no idea how you would identify that
<gmaxwell>
I should see how much data I can store on QR-like codes engraved in steel plates.
<phantomcircuit>
i've definitely had ssds which failed with soft errors that magically were fine when i zero'd the drive
<phantomcircuit>
gmaxwell: iirc a few kb
<gmaxwell>
well also rewriting slowly chews write endurance, ... presuambly better than letting the data rot.
<phantomcircuit>
gmaxwell: depends on what the drives for
<sipa>
db_dump + simple python tool should be able to decrypt any key records easily
<phantomcircuit>
windows gaming computer? i probably don't want background refresh on
<sipa>
i guess the key strengthening is somehow nonstandard
SpellChecker_ has quit [Remote host closed the connection]
SpellChecker has joined #bitcoin-core-dev
<phantomcircuit>
sipa: isn't it just sha256 in a loop?
<phantomcircuit>
no it's pbkdfjfj2222 in a loop
<phantomcircuit>
easy to replicate in python (if slowly)
<sipa>
phantomcircuit: yes, sure, so it maybe 5 minutes of work instead of 1 :)
Christoph_ has quit [Quit: Christoph_]
<phantomcircuit>
sipa: iirc db_dump even with -R basically never recovers any data, but rather 99% of the time just zeros things until it loads
<gmaxwell>
phantomcircuit: on engraving, maybe not as a QR code but I can store a lot more than that, just not sure how much. the engraver is accurate to a couple micron. so getting a raw capacity the order of 80mbits/inch sounds credible.
<phantomcircuit>
gmaxwell: yeah but reading it sounds annoying
<sipa>
that's inch, not inch^2 ?
<gmaxwell>
in^2
<phantomcircuit>
sipa: gmaxwell: is a one dimensional being
<gmaxwell>
phantomcircuit: well reading is just an x/y table microscope, but my thought is you could backup data w/ tape + plate and the plate is just the backup. Write only, hopefully.
<sipa>
engineers approximate everything with linear functions
<gmaxwell>
phantomcircuit: there was some project for forever data storage that used laser punched metal tape. but they seemed to get obsessed with people being able to read photographs by just looking at the tape and stuff like that and more or less completely eschewed error correcting codes, which seems insane to me. (esp because you could just put the spec for the error correcting code on every tape).
<phantomcircuit>
gmaxwell: but what if 100000 years in the future the tape is being read by a primitive caveman and not some super advanced galactic mannnnn
<phantomcircuit>
the nuclear "waste" disposal signs remind me of that
<phantomcircuit>
like how is it even plausible that we need to explain nuclear material to people 100 years from now unless we nuked everything and then who cares about where the waste is
<gmaxwell>
yeah thats their logic. otoh if caveman can't figure out how decode the error correcting code that is described on the tape, whats he gonna do with billions of bits of information anyways?
<gmaxwell>
phantomcircuit: the waste stuff though is targeting more like 10k-100k years. seems reasonable that the risk might not be understood.
<achow101>
in my tool so far, I have dumping encrypted private keys already
<achow101>
the annoying part is the whole bdb thing...
sliv3r__ has quit [Quit: ZNC 1.8.2+deb3.1+deb12u1 - https://znc.in]
sliv3r__ has joined #bitcoin-core-dev
<achow101>
maybe I'll drop the bdb dependency and reimplement parsing once again, but the point was to be able to insert whatever into a wallet file
<sipa>
achow101: don't you have a python reimplementation of bdb already?
<achow101>
sipa: in the functional tests in old branches, although that is of course tailored to our functional tests
<gmaxwell>
For real ultimate recovery I think you kinda want to do both your own parsing and BDB, just in case one or the other is more successful. Partially corrupted file seem remarkably common for some reason. like I found I had corrupted files inside zips that were absolutely not corrupted, and I had no idea how they got corrupted.
<achow101>
gmaxwell: it turns out that having bdb itself on a system in a way that people can use from a python script is non-trivial, especially when you need a specific version
<phantomcircuit>
gmaxwell: if in 10k years we haven't consumed all the nuclear "waste" then humanity has long since been extinct
<gmaxwell>
achow101: hm, for reading though the most current version works on old versions, it just makes files that are unreadable in old versions-- no?
<phantomcircuit>
gmaxwell: bdb will write log files into the ~/.bitcoin directory, if you happen to copy the .dat file when it's doing that you'll get corruption that's fixable with db_dump -R, but which iirc basically deletes records until the database is consistent
<gmaxwell>
phantomcircuit: nah, our civilization could be extinct, but some amish surely will survive the end of the world. and if not them, then the dolphin people. :P
<achow101>
gmaxwell: yeah, but I don't think you can guarantee it won't write anything even when opening as readonly
<gmaxwell>
achow101: right so tool should copy the file, then open it, and throw out its temp copy.
<achow101>
pfft, working on copies, who does that :p
<gmaxwell>
good to do anyways even with the older library version because the user may not have been informed enough to make copies before they start trying to recover stuff.
<gmaxwell>
(and god knows wtf bdb will do to a corrupted file)
<gmaxwell>
I'm pretty sure I lost a few bitcoin a decade ago by BDB just deciding to strip everything out of some corrupted file.
<sipa>
good old savagewallet RPC?
<gmaxwell>
not entirely sure, as I had a wallet that all my records and even shell history say should have coins, but it contained no key records when I went to dig into it.
<achow101>
gmaxwell: it could be in the log files
<gmaxwell>
I may have hit a startup error, just did a salvage to continue and not thought about it till later.
<gmaxwell>
achow101: this was a decade ago, and I didn't notice the wallet had been key-stripped until a llloong time later.
<gmaxwell>
it also wasn't worth that much at the time, so not a huge deal. just mentioned it as a reminder to not trust bdb to not screw things up.
<phantomcircuit>
gmaxwell: i feel like bdb was designed to break randomly to further sell oracledb licences
<gmaxwell>
I think just making any database that works reliably on computers is hard. there are very few guarentees on what makes it to disk at any given time.
S3RK_ has quit [Ping timeout: 244 seconds]
S3RK has joined #bitcoin-core-dev
nool has joined #bitcoin-core-dev
nool has quit [Quit: Leaving]
aprilwall_ has joined #bitcoin-core-dev
aprilwall_ has quit [Client Quit]
brunoerg has joined #bitcoin-core-dev
brunoerg has quit [Remote host closed the connection]