Megadrive SRM Saving issue..getting closer to solving..

[gliitch]

Hi,

I have, along with Wannado have  been doing some research into why the SRM files for Megadrive games are "disappearing"



It's possibly due to the emulators saving the games in different sectors of the SRM files. So when you load your save file in a different emulator such as Kega, the emulator will search for where it thinks the save data should be but instead is greeted by 00x00 instead of whatever the save information should be. We just need to find the correct sector & stop the retrode from replacing the save file with a new blank file.

Here's what Wannado says: I'm guessing the following: In the Retrode file, the data repeats after 16 kiB. This is also known as a mirror, because both copies of the data come from the same location in memory.

The emu loads the whole 32 kiB, but the game only modifies the first 12 kiB. The emu still saves the whole 32 kiB, because it cannot be sure that truncating the file would be safe. The result is that the save on the cartridge is overwritten twice: First with the new data (first 12 kiB), then with the "second copy" of the old data starting at the 16 kiB offset. The latter "restores" the old save. (The 4 kiB pieces at the 12 kiB and 28 kiB offsets are probably lost as they are beyond the end of cartridge memory.)

I have tested it with Kega, and found that the SRM files differ depending on the type of emulator you use.  I have a feeling that the Retrode itself wipes out your original save upon connection & replaces it with a blank save instead which doesn't help matters.

However, if you create a save with a specific emulator, say like KEGA save that, copy it over. let the Retrode wipe out your save, copy back the kega save, then reload the emu. The previous non existing save will reappear. All we have to do as Wannado puts it is detect the correct save size for each game.

I think that with Wannado & help from the community we are getting closer to solving this issue.

Upon further inspection, my hunch that the retrode does indeed create new saves was correct.
If you look at the dates the Retrode reports, they change from (fresh save)  4/7/2016 >  upon disconnection /re connection 20/11/1990 along with an altered time of 23:00





 

[Matthias_H]

Plausible explanation, except that the Retrode doesn't "create new saves" - if anything, the OS does. For various reasons, the Retrode's entire file system is made up on the fly, and a change in the file date doesn't have anything to do with the question whether it's an old or new file.

[gliitch]

If thats the case, then surely there must be a way to stop the OS from creating a new file. As ive noticed with a hex editor, upon re connection the top of the file reports all FF's which from what it looks like is that the OS has blanked out some important information.

Where as a file which i replaced & i know that works, it has FF 00 FF 00 FF 50 FF 48 FF 41 FF 4E FF 54 FF 41 FF 53 FF 59 FF 20 FF 53 FF 54 FF 41 FF 52 FF 33 FF 65 FF D6 FF 49 FF 00 FF 52 FF 68 FF 79 FF 73 FF FC FF 00 FF 29 FF 00 FF 00 FF 00 FF 07 FF 00 FF 00 FF 04 FF 8F FF 01 FF 10 FF 01 FF 40 FF 00 FF 00 FF 01 FF 80 FF 01 FF 00 FF 00 FF 80 FF 02 FF 00 FF 02 FF 80 FF 00 FF 04 FF 00 FF 02 FF 00 FF 00 FF 00 FF 06 FF 00 FF 08 FF 29 FF 48 FF (which comes from Phantasy Star IV) when saved with Kega Fusion.

When this is loaded, up comes the save file. I don't have a way to edit the SRM, to test if this is of any use. Also if you look that in a hex editor, it comes up with jumbled up lettering.

When i spoke to Wannado, he did mention that i had a point about the size of the save file not being the right size. & that could well pose an issue as to why the saves are not working.  Either way, i think we are getting closer to the retrode being cross platform emulator / megadrive.

[Wannado]

My statement that gliitch quoted in the OP was a response to a PM not quoted here. The numbers given may be wrong, but the general idea may still be right.

In the meantime, gliitch has sent me a few sample save files (thanks!), and I have taken a look at them. Those save files have been created by different means, but belong to the same game (seemingly Phantasy Star 3 - Generations Of Doom).

1.
From "everdrive":
I guess this file has been created playing the game using an Everdrive plugged into a MegaDrive, and retrieved by reading the Everdrive's storage media with a regular card reader. Hence there was no interference from the Retrode.
@gliitch: Is that correct?

The file size is 12 kiB, but the first of every two bytes has the value FF. Removing those FF bytes yields 6 kiB of data resembling the other save files provided:
* 2 kiB savegame.
* 2 kiB unused memory (all FF).
* 2 kiB savegame identical to the first one - @gliitch: Did you store the same game in two slots (I'd have), or might this be some kind of mirror?

Note: I guess that this is followed by another 2 kiB of unused memory, for a total of 8 kiB of memory, but the Everdrive truncates the file to the size that is actually used.


2.
From "megadrive":
I guess this file has been created using the original game cartridge in a MegaDrive, and retrieved using the Retrode.
@gliitch: Is that correct?

The file size is 16 kiB minus 1 byte (why?). The data structure is:
* 2 kiB savegame.
* 2 kiB unused memory "A" (filled with alternating patterns of FF FF ... 00 00 ... disturbed by very few different, "random" values).
* 2 kiB savegame (identical).
* 2 kiB unused memory "B" (like above, but the "random" values and their locations are different).
* 2 kiB savegame (identical).
* 2 kiB unused memory identical to "A".
* 2 kiB savegame (identical).
* 2 kiB - 1 byte unused memory identical to "B", except that the last byte is missing.


3.
From "retrode":
I guess this file has been created using an emulator (Kega Fusion), the Retrode, and the original game cartridge.
@gliitch: Is that correct?

The file size is 16 kiB minus 1 byte. The data structure is:
* 8 kiB filled with alternating patterns of FF FF ... 00 00 ... disturbed by very few different values.
* 8 kiB - 1 byte identical to the first 8 kiB, except that the last byte is missing.

So, compared to the "megadrive" file, the "retrode" file has a similar structure, but the save slots seem to be empty.


Based on the above, my current, still flawed theory is:

The game has 8 kiB of memory used for two save slots. Or maybe four save slots, and gliitch likes to use only the first and third of them, for a game and a backup copy. Or the game itself duplicates the save slot(s) because there is such an abundance of memory, and if one copy is corrupted, the other may still be valid.

For some reason, the Retrode presents a 16 kiB save file. The first 8 kiB (16 sectors) of the file are mapped to the save memory as intended. But when the Retrode tries to access the nonexistent second 8 kiB of memory, the cartridge ignores the highest 1-bit of the address and again returns (when reading) or modifies (when writing) the data in the first 8 kiB. So the second 8 kiB of the file are mapped to the same memory as the first ("mirror").

See also my statement in the OP, with 32 kiB / 12 kiB replaced by 16 kiB / 6 or 8 kiB.

The flaw in this theory is: Above, gliitch said that the Retrode "wipes out" the save, replacing it with a "blank" one (like 3. above). But in my theory, the save should only be reverted to its previous state, not vanish completely.

This would be consistent if the "previous state" was "blank", but I have no idea how gliitch could have created such a state after playing the game on the MegaDrive. Note the imperfect FF FF ... 00 00 ... patterns: if the game had intentionally cleared the slots, it would likely have filled them with all 00 (or all FF) instead.

I don't think that the battery is dead, because gliitch would likely have noticed that, and creating the "megadrive" file (2. above) would have failed then.

I don't know how the transition from "console power" to "battery power" works (or if there is a transition at all), but maybe the Retrode is controlling it wrongly, causing a short blackout in the save RAM.


Any ideas / comments?

[gliitch]

@Wannado, to save space on this thread, all of the question that you have raised, you are indeed correct. I am not too well versed in data structures. I will help where i  can.  It certainly seems as though you have a far better understanding of these values and being able to create bug fixes (take the last firmware update for example) then i do, it would be awesome to learn a few new things

[Nori]

Glad there's still work ongoing on this issue. It's one of the biggest uncorrected flaws of the Retrode IMO. Basically I believe it's impossible to overwrite Genesis SRM's that have been altered through an emulator. The cart will always retain it's previous SRM unless moving back an SRM that came off a cart and hasn't been altered in an emu. The other problem is many Genesis emus use different save files. I've seen 16kb, 32kb and I believe even 64kb's with some emus. Sometimes they work between each other and convert the file sizes to the newly used emu. In other words it's a bit of a mess. :/

[KingMike]

I haven't looked it up, does Retrode support Phantasy Star IV correctly in the first place?

As I have read, PS4 uses a mapper where the first 2MB of ROM is always mapped in, and then the last 2MB of Genesis cartridge space can map in either of the last 1MB ROM or the SRAM. (and I heard that at one point the game does a copy-protection check by mapping in ROM and attempting to write to it)

[Nori]

I believe it works but somehow it deleted my save file in slot 1 and 3. Fortunately all three were late game saves past a ton of grinding but since I hadn't played the game in almost 15 years I can't remember which was the best slot. Anyway all I had left was slot 2 after copying the save. I didn't try saving in 1 and 3 to try again. If anybody has please share the results.

[Wannado]

I found a possible explanation for the issue: The cartridge's address bus does not have a line for the least significant bit (LSB) of the byte address. Instead, its data bus is 16 bits wide, so that a two-byte "word" can be read at once.

Hence the Retrode has to divide the byte addresses by two to form word addresses, and use the byte address LSB to select which half of the data bus to read from/write to. Also, some cartridges connect only one half of the data bus to SRAM, and emulators disagree about how to handle this. That's why there is the segaSram16bit setting in RETRODE.CFG.

And the bug might be here: The ROM and SRAM sizes are computed from the respective start and end addresses given in the ROM header. I think that all four addresses are byte addresses. But the Retrode firmware seems to treat the SRAM limits as word addresses, so that the computed SRAM size is two times the real SRAM size.

The next release, v0.20 beta, will contain an attempt to detect what the unit of the SRAM limits in the ROM header really is, and to compute the size accordingly. (The Retrode expects the SRAM limits to fall within in the byte address range 0x200000 to 0x20FFFF, which equals the word address range 0x100000 to 0x107FFF.)

Update 2016-10-29: Logfiles gliitch sent me indicate that the SRAM limits are stored in the header as byte addresses, just like the ROM limits.

[Nori]

Wow I had ended up thinking we'd never actually get this issue resolved someday. Great work Wannado, as always you're the key to keeping this awesome project alive!

[Wannado]

If only I was keeping it alive faster.

I don't have any MegaDrive cartridges to test if this really resolves the whole issue. But at least it is consistent with my theory about the mirroring.

Just in case, the release will also include a firmware variant that produces a logfile with information that might help in the analysis of this issue.

[Nori]

Great. Can't wait to try it out. 

[Wannado]

I sent it to Matthias two weeks ago, but maybe it was lost or he just didn't have time to upload it, so to speed things up, I'm posting it here (see attachments).

See the enclosed readme.txt for details.

Note that I can only perform few tests on each release. In this case, I didn't even have the means to verify that the bug fixes had their intended effects. I only tested that a few other things still work as before.

I couldn't test some of the log messages I added, so the logfile variant might freeze or otherwise misbehave.

Update 2016-10-29: Meanwhile, gliitch and Pickle have performed some tests, see reply #14.
Update 2016-11-14: WARNING: The computed SRAM size is still wrong, see reply #19.

[MasterOfPuppets]

Thank you for the new version!

[Wannado]

gliitch has performed quite a number of tests of v0.20 beta with Phantasy Star III, Phantasy Star IV, and Shining Force.

Thanks a lot, gliitch!

He reports that the savegame file (*.srm) mostly works (but see restriction below):

• The SRAM size seems to be detected correctly now (see also Reply #8 above). Update: It's still wrong in many cases, see reply #19.
  
• The file can be copied to and from the cartridge properly (see requirements below).
• Direct access from emulators (including on Android) also works (see below).

Requirements:

• The game/console region may be relevant for savegame compatibility. This is an issue of the games/emulators, not the Retrode.
  
• It only works with firmware v0.20 beta or later.
• You must set "[segaSram16bit]" in RETRODE.CFG to 0 or 1 depending on the format used by the emulator / of the file to be copied.
  I guess a simple rule of thumb is: If the file sizes are different by a factor of 2, toggle the setting. I recommend that you copy the file from the cartridge as a backup before and after toggling the setting. The smaller backup might be useless - keep both.
• By default, the Retrode's savegame file is write protected. To disable the write protection and be able to modify the on-cartridge data, you must set "[sramReadonly] 0" in RETRODE.CFG.

Restriction:

• There still seems to be an issue with the third save slot of Phantasy Star IV and Shining Force. It might be wiped or inaccessible.

Thanks also go to Pickle!

Pickle has tested the combination of "[forceSize] 8" and "[forceSystem] GG" with v0.20 beta and reports that it worked. (Said combination indicates an 8 megabit Game Gear cartridge.)