Makes it possible to browse eMMC partitions and dump data from them.
Since we're manually parsing FAT partitions, reading files protected by the FS sysmodule at runtime is possible.
Takes care of retrieving a FsStorage object for any FAT eMMC BIS partition, mounting it via FatFs and creating a virtual devoptab device that can be used to carry out FS operations. All write operations have been stubbed, disabled or ifdef'd out of the code.
Other changes include:
* cert: update code to use the new BIS storage interface.
* defines: remove BIS_SYSTEM_PARTITION_MOUNT_NAME macro.
* devoptab: slightly improve macros.
* devoptab: add operation table for FatFs devices.
* devoptab: add rodev_fstat().
* devoptab: add devoptabMountFatFsDevice().
* fatfs: update diskio code to use the new BIS storage interface.
* fatfs: update configuration.
* save: update code to use regular C I/O calls instead of FatFs calls.
* tik: update code to use the new BIS storage interface.
* utils: remove eMMC BIS System partition (un)mount code.
It will be used to dump system updates by parsing content meta info records from the SystemUpdate CNMT while also validating all content hashes and signatures.
A hardcoded table with HOS version numbers and master key indexes is now used to determine the HOS key generation at runtime, whenever possible. This allows the application to more accurately determine the key generation that's actually required by the console it's running on.
Most parts of the code that relied on the Atmosphère key generation value have been updated to use the HOS key generation value instead. If the HOS version is too high/unknown, the code will fallback to the Atmosphère key generation value.
Furthermore, if the HOS key generation value is lower than our last known key generation, the code will now try to look for the highest available master key it can use to derive all lower master keys, beginning with the last known master key and ending with the master key that matches the HOS key generation value. Previous behavior only checked the availability of the master key that matched the Atmosphère key generation, which isn't completely reliable nor accurate.
If this process fails, current master key derivation will be carried out as a last resort, which wasn't being done either under this specific scenario.
Other changes include:
* keys: add keysGetHorizonOsKeyGeneration().
* keys: move current master key derivation logic into its own function, keysDeriveCurrentMasterKey(), which is now used if both Atmosphère and HOS and up-to-date, or if a lower master key is required (as a last resort method).
Other changes include:
* title: update titleGetContentInfosByGameCardContentMetaContext() to make it write the forged NcmContentInfo entry for the Meta NCA at the start of the returned buffer (matches ncm output).
Updates pfsAddEntryInformationToImageContext() to handle name table padding and alignment on its own. As a result, the logic from pfsWriteImageContextHeaderToMemoryBuffer() has been heavily simplified.
Changes include:
* nxdt_utils: add utilsEscapeCharacters().
* title: add titleGenerateTitleRecordsCsv().
* title: move core logic from titleGetUserApplicationData() into _titleGetUserApplicationData(). titleGetUserApplicationData() now only takes care of duplicating the retrieved data.
* title: update titleGetContentInfosByGameCardContentMetaContext() to make it write the forged NcmContentInfo entry for the Meta NCA at the end of the returned buffer.
* title: fix a bug in titleRefreshGameCardTitleInfo() that prevented a title info's metadata pointer to be updated after retrieving application metadata via ns.
* title: update titleGenerateGameCardApplicationMetadataArray() to add a logfile warning if an application entry doesn't have a valid application metadata pointer.
* title: make _titleGenerateGameCardFileName() a bit easier to read.
Makes it possible to generate TitleInfo and TitleApplicationMetadata elements for gamecards that can't be used with ncm (e.g. Kiosk / Quest gamecards under retail, non-Quest units). This is achieved by:
1. Retrieving a Hash FS context for the gamecard's secure partition.
2. Initializing NCA and CNMT contexts for all of the Meta NCAs within the secure partition.
3. Manually generating NcmContentMetaKey and NcmContentInfo elements per each CNMT context.
4. Creating TitleInfo elements using the generated NcmContentMetaKey and NcmContentInfo elements.
5. Initializing NCA and NACP contexts for the base (or update) Control NCAs within the secure partition.
6. Manually generating a NsApplicationControlData element per each NACP context.
7. Creating TitleApplicationMetadata elements using the generated NsApplicationControlData elements.
Afterwards, gamecard title/content enumeration and all other features that rely on it (e.g. NSP, NCA, NCA FS) work as expected. Please note that this process is only carried out if regular title storage initialization fails.
Other changes include:
* title: reorder code.
* title: add TitleGameCardContentMetaContext struct.
* title: rename titleGetInfoFromStorageByTitleId() -> titleGetTitleInfoEntryFromStorageByTitleId().
* title: add titleInitializeGameCardTitleStorageByHashFileSystem().
* title: rename titleGenerateDummySystemMetadataEntry() -> titleGetSystemMetadataEntry().
* title: rename titleRetrieveUserApplicationMetadataByTitleId() -> titleGenerateUserMetadataEntryFromNs().
* title: move ns logic from titleRetrieveUserApplicationMetadataByTitleId() into a new function: titleGetApplicationControlDataFromNs().
* title: add titleGenerateUserMetadataEntryFromControlNca().
* title: add titleGetApplicationControlDataFromControlNca().
* title: add titleInitializeUserMetadataEntryFromControlData().
* title: add titleGenerateTitleInfoEntriesByHashFileSystemForGameCardTitleStorage().
* title: move TitleInfo generation logic from titleGenerateTitleInfoEntriesForTitleStorage() into a new function: titleGenerateTitleInfoEntry().
* title: add titleInitializeTitleInfoApplicationMetadataFromControlNca().
* title: add titleGetGameCardContentMetaContexts().
* title: add titleFreeGameCardContentMetaContexts().
* title: add titleGetContentInfosByGameCardContentMetaContext().
* title: rename _titleGetInfoFromStorageByTitleId() -> _titleGetTitleInfoEntryFromStorageByTitleId().
* title: rename titleSystemTitleMetadataEntrySortFunction() -> titleSystemMetadataSortFunction().
* title: rename titleUserApplicationMetadataEntrySortFunction() -> titleUserMetadataSortFunction().
* title: rename titleInfoEntrySortFunction() -> titleInfoSortFunction().
* title: add titleGameCardContentMetaContextSortFunction().
* codebase: update to reflect the rest of the changes.
Other changes include:
* cnmt, nca: handle NcmContentId elements in a more uniform way.
* gamecard: add some comments to the GameCardFlags enum.
* nxdt_log: make sure log messages are horizontally aligned.
Changes include:
* title: define a global string array for the pre-generated gamecard filenames.
* title: move gamecard filename generation logic into a new function, _titleGenerateGameCardFileName().
* title: update background gamecard thread logic to make it generate gamecard filenames on any status changes.
* title: update titleGenerateFileName() to make it return a duplicate of the gamecard filename requested by the caller.
Changes include:
* mem: change all references to `Rodata` to `RoData`.
* nso: change all references to `Rodata` to `RoData`.
* nso: update `NsoModStart` struct.
* nso: rename `NsoModuleInfo` struct to `NsoModulePath`.
* nso: add `NsoRoDataStart` struct.
* nso: update logic in both nsoIsNnSdkVersionWithinSegment() and nsoGetNnSdkVersion() functions to work entirely with memory-based offsets.
* nso: rename nsoGetModuleInfoName() to nsoGetModulePath().
* nso: update logic in nsoInitializeContext() to always validate and use the NsoNnSdkVersion block offset from the NsoModStart block.
titleGetApplicationMetadataEntries() and titleGetGameCardApplicationMetadataEntries() will now return dynamically allocated copies of internal pre-filtered / pre-processed arrays, which are generated using the background gamecard thread. This results in less overhead for any potential calls to these functions.
Other changes include:
* title: rename TitleGameCardApplicationMetadataEntry -> TitleGameCardApplicationMetadata.
* title: add `has_patch` field to TitleGameCardApplicationMetadata struct.
* title: declare internal TitleApplicationMetadata arrays to hold pre-filtered application metadata.
* title: declare internal TitleGameCardApplicationMetadata array to hold pre-processed gamecard application metadata.
* title: move filtering logic from titleGetApplicationMetadataEntries() to a new function: titleGenerateFilteredApplicationMetadataPointerArray().
* title: move processing logic from titleGetGameCardApplicationMetadataEntries() to a new function: titleGenerateGameCardApplicationMetadataArray().
* title: rename titleGetPatchVersionString() -> titleGetDisplayVersionString().
* title: add extra debug log messages to some functions.
* title: update titleFreeApplicationMetadata() to also free the new internal metadata arrays.
* title: update background thread logic in titleGameCardInfoThreadFunc() to also regenerate the pre-filtered application metadata and gamecard application metadata arrays right after a successful call to titleRefreshGameCardTitleInfo().
* title: update titleGetDisplayVersionString() to also support base application titles.
* title: simplify string generation logic in titleGenerateGameCardFileName() by using the cached gamecard application metadata array.
* GameCardStatusTask: add GetGameCardStatus() method.
* GameCardTab: fix callback argument type in class constructor.
* GameCardTab: update ProcessGameCardStatus() to block user inputs while processing the new gamecard status.
* RootView: add GetGameCardStatus() method.
* StatusInfoTask: turn IsInternetConnectionAvailable() into an inline method.
* TitleMetadataTask: turn GetApplicationMetadata() into an inline method.
* TitleMetadataTask: move debug log messages around.
* TitlesTab: update PopulateList() to block user inputs while updating the titles list.
* UmsTask: turn GetUmsDevices() into an inline method.
* UsbHostTask: turn GetUsbHostSpeed() into an inline method.
ftruncate() may severely delay the start of a dump process for certain UMS devices, and the write speed differences when using the SD card seem to be negligible.
Other changes include:
* FileWriter: add log messages throughout all class methods.
* log: update generic log macros to use __PRETTY_FUNCTION__ instead of __func__.
* log: add logFormatFunctionName() function, which takes an input __PRETTY_FUNCTION__ string and parses the info we need from it.
* log: update _logWriteFormattedStringToLogFile() and logWriteFormattedStringToBuffer() functions to reflect the other changes.
Other changes include:
* config: remove ConfigChecksumLookupMethod enum.
* config: remove "gamecard/checksum_lookup_method" integer field.
* config: add "gamecard/lookup_checksum" boolean field.
* config: update default configuration file.
* defines: remove all NSWDB macros.
* GameCardImageDumpTask: update class to reflect the changes made to the configuration interface.
* GameCardImageDumpOptionsFrame: update class to reflect the changes made to the configuration interface.
* OptionsTab: remove OptionsTabUpdateFileDialog class.
* OptionsTab: remove "Update NSWDB XML" element.
* i18n: update localization files to reflect the rest of the changes.
Other changes include:
* gamecard: set GameCardStatus_Processing as the current gamecard status before calling gamecardLoadInfo().
* workflow: try to fix the missing commit tag issue.
The directory layout is partially based on the C++ namespaces we're currently using.
Other changes include:
* devoptab: move directory into "core".
* fatfs: move directory into "core".
* GameCardTab: move portions of logic from PopulateList() into their own methods.
* GameCardTab: use a macro to generate the properties table.
* GameCardTab: use a macro to add ListItem elements.
* GameCardTab: update AddApplicationMetadataItems() method to also display the number of DLCs available in the inserted gamecard for each application whenever possible.
* Makefile: remove all extra entries from the INCLUDES variable.
* nxdt_includes: move HOS version structs into their own header file.
* tasks: move code for each individual task into its own file(s).
* title: update titleGetGameCardApplicationMetadataEntries() to also count the number of DLCs available in the inserted gamecard for any given base application.
* title: reorder gamecard application metadata entries by name before returning the buffer in titleGetGameCardApplicationMetadataEntries().
DataTransferTaskFrame is a template class that's derived from brls::AppletFrame, which automatically starts a background task using an internal object belonging to a class derived from DataTransferTask. A DataTransferProgressDisplay view is used to show progress updates. If the background task hits an error, the class takes care of switching to an ErrorFrame view with the corresponding error message.
GameCardImageDumpTaskFrame is a derived class of DataTransferTaskFrame that uses a GameCardImageDumpTask object as its background task. In layman's terms, this provides a way to fully dump gamecard images using the new UI.
DataTransferTaskFrame depends on the newly added is_base_template helper from goneskiing to check if the class for the provided task is derived from DataTransferTask.
Other changes include:
* DataTransferProgressDisplay: rename setProgress() method to SetProgress().
* DataTransferTask: move post-task-execution code into its own new private method, PostExecutionCallback(), and update both OnCancelled() and OnPostExecute() callbacks to invoke it.
* DataTransferTask: update OnProgressUpdate() to allow sending a last progress update to all event subscribers even if the background task was cancelled.
* DataTransferTask: update OnProgressUpdate() to allow sending a first progress update if no data has been transferred but the total transfer size is already known.
* DataTransferTask: update OnProgressUpdate() to avoid calculating the ETA if the speed isn't greater than 0.
* DumpOptionsFrame: remove UpdateOutputStorages() method.
* DumpOptionsFrame: update class to use the cached output storage value from our RootView.
* DumpOptionsFrame: add GenerateOutputStoragesVector() method, which is used to avoid setting dummy options while initializing the output storages SelectListItem.
* DumpOptionsFrame: update UMS task callback to add the rest of the leftover logic from UpdateOutputStorages().
* DumpOptionsFrame: update RegisterButtonListener() to use a wrapper callback around the user-provided callback to check if the USB host was selected as the output storage but no USB host connection is available.
* ErrorFrame: use const references for all input string arguments.
* FileWriter: fix a localization stirng name typo.
* FileWriter: fix an exception that was previously being thrown by a fmt::format() call because of a wrong format specifier.
* FocusableItem: add a static assert to check if the provided ViewType is derived from brls::View.
* gamecard: redefine global gamecard status variable as an atomic unsigned 8-bit integer, which fixes a "status-hopping" issue previously experienced by repeating tasks running under other threads that periodically call gamecardGetStatus().
* GameCardImageDumpOptionsFrame: define GAMECARD_TOGGLE_ITEM macro, which is used to initialize all ToggleListItem elements from the view.
* GameCardImageDumpOptionsFrame: update button callback to push a GameCardImageDumpTaskFrame view onto the borealis stack.
* GameCardImageDumpTask: move class into its own header and module files.
* GameCardImageDumpTask: update class to also take in the checksum lookup method (not yet implemented).
* GameCardImageDumpTask: update class to send its first progress update as soon as the gamecard image size is known.
* GameCardImageDumpTask: update class to avoid returning a string if the task was cancelled -- DataTransferTaskFrame offers logic to display the appropiate cancel message on its own.
* GameCardTab: update PopulateList() method to display the new version information available in TitleGameCardApplicationMetadataEntry elements as part of the generated TitlesTabItem objects.
* i18n: add new localization strings.
* OptionsTab: update background task callback logic to handle task cancellation, reflecting the changes made to DataTransferTask.
* OptionsTab: reflect changes made to DataTransferProgressDisplay.
* RootView: cache the currently selected output storage value at all times, which is propagated throughout different parts of the UI. Getter and setter helpers have been added to operate with this value.
* RootView: add GetUsbHostSpeed() helper, which can be used by child views to retrieve the USB host speed on demand.
* RootView: update UMS task callback to automatically reset the cached output storage value back to the SD card if a UMS device was previously selected.
* title: define TitleGameCardApplicationMetadataEntry struct, which also holds version-specific information retrieved from the gamecard titles.
* title: refactor titleGetGameCardApplicationMetadataEntries() to return a dynamically allocated array of TitleGameCardApplicationMetadataEntry elements.
* usb: redefine global endpoint max packet size variable as an atomic unsigned 16-bit integer, which fixes a "status-hopping" issue previously experienced by repeating tasks running under other threads that periodically call usbIsReady().
* UsbHostTask: add GetUsbHostSpeed() method.
GameCardImageDumpTask is a derived class of DataTransferTask, and it's designed to dump a gamecard image using the options selected by the user (which must be passed from a GameCardImageDumpOptionsFrame object). It uses std::optional<std::string> as its return type -- the idea behind this is to return error strings that may later be displayed by an ErrorFrame during the dump process (views not yet implemented).
FileWriter is a class that encapsulates write operations to different storage mediums (SD card, USB host and UMS devices), based on the provided input path. It is used by GameCardImageDumpTask to painlessly write data to the right output storage without explicitly having to implement multiple code paths for all storage types as part of the actual dump code. Furthermore, FileWriter also supports writing split files to FAT-formatted UMS devices if an output file is >= 4 GiB -- part file handling is completely abstracted away from any callers.
Other changes include:
* AsyncTask: rename all class methods to use PascalCase naming.
* AsyncTask: rename get() -> GetResult().
* DataTransferTask: reflect the changes made to AsyncTask.
* DataTransferTask: pause the RepeatingTask right after LoopCallback() returns true instead of pausing it in the cancel/post-execute callbacks.
* DataTransferTask: add private FormatTimeString() method.
* DataTransferTask: remove superfluous override for DoInBackground() -- classes derived from DataTransferTask must provide it on their own, anyway.
* DataTransferTask: add public GetDurationString() method.
* defines: update FAT32_FILESIZE_LIMIT macro to use UINT32_MAX.
* defines: add CONCATENATION_FILE_PART_SIZE macro (used by the new FileWriter class).
* DownloadTask: reflect the changes made to AsyncTask.
* DumpOptionsFrame: file extension is no longer stored as a class member, nor required by the class constructor.
* DumpOptionsFrame: change the return type for GetOutputFilePath() to bool. The method now saves its output to a variable passed by reference.
* GameCardImageDumpOptionsFrame: reflect the changes made to DumpOptionsFrame.
* i18n: update localization strings where applicable.
* nxdt_utils: fix a potential buffer overflow in utilsGetFileSystemStatsByPath().
* OptionsTab: reflect the changes made to AsyncTask.
* usb: add const qualifier to the input buffer required by usbSendFileData().
* usb: add const qualifier to the input buffer required by usbSendNspHeader().
Other changes include:
* DumpOptionsFrame: update contructors to also take a base output path string.
* GameCardImageDumpOptionsFrame: simplify constructor by letting it take care of retrieving the title on its own.
* nxdt_utils: remove utilsCreateOutputDirectories() function -- we'll be using utilsCreateDirectoryTree() anyway, so it's okay.
These changes include:
* gamecard: add GAMECARD_HEADER2_OFFSET and GAMECARD_HEADER2_CERT_OFFSET macros.
* gamecard: rename GAMECARD_CERTIFICATE_OFFSET macro to GAMECARD_CERT_OFFSET.
* gamecard: add GameCardFlags2 enum.
* gamecard: update GameCardHeader struct to reflect the new Flags2 field.
* gamecard: add GameCardHeader2, GameCardHeader2EncryptedData and GameCardHeader2Certificate structs.
* gamecard: update gamecardReadHeader() to check if the `Has10Certificate` flag is set in the gamecard header. If it is, the function will read and log both Header2 and Header2Certificate areas, and it will also attempt to verify the RSA signature from the Header2 area.
* gamecard: if a Header2 area is detected, an error will always be triggered for the time being. If anyone comes across this error, please get in touch me.
* gamecard, fs_ext: fix signature field comments.
Adds the DataTransferTask and DataTransferProgressDisplay classes, which use repurposed logic from the DownloadTask and EtaProgressDisplay classes, respectively. The EtaProgressDisplay class and EtaProgressInfo struct have been removed.
Other changes include:
* DownloadTask: heavily simplified logic by using DataTransferTask as the base class.
* DumpOptionsFrame: implement some additional/experimental changes.
* OptionsTab: update logic to use the new DataTransferProgressDisplay class.
Other changes include:
* gamecard: rename gamecardGetDecryptedCardInfoArea() -> gamecardGetPlaintextCardInfoArea().
* FocusableItem, FocusableTable: add positional arguments for the FocusableItem class constructor.
Removes all references to Compressed storages being used as substorages for Indirect storages.
I should have taken care of this in 0f1055c84e, but I guess I just forgot.
* codebase: remove all references to Lockpick / Lockpick_RCM.
* keys: retrieve Atmosphère's key generation in keysLoadKeyset().
* keys: replace all "key_gen_val" references with "mkey_index".
* keys: move latest master key checks in keysReadKeysFromFile() to keysDeriveMasterKeys().
* keys: update latest master key checks to determine if nxdumptool's hardcoded master key vectors are outdated, using the key generation value from Atmosphère. If the master key vectors are outdated, and the newer master key(s) are not available, an error will be displayed.
* nxdt_utils: update logic in utilsReplaceIllegalCharacters() to replace consecutive illegal characters with a single underscore.
* nxdt_utils: move servicesGetExosphereApiVersion() to nxdt_utils as utilsGetExosphereApiVersion().
* nxdt_utils: define internal UtilsExosphereApiVersion struct, which is used to parse the output from utilsGetExosphereApiVersion().
* nxdt_utils: add utilsGetAtmosphereVersion(), utilsGetAtmosphereKeyGeneration() and utilsGetAtmosphereTargetFirmware() functions.
* services: remove servicesGetExosphereApiVersion().
* services: Atmosphère version is now retrieved via utilsGetAtmosphereVersion().
Other changes include:
* poc: fix switching to a different title entry via L/R/ZL/ZR if the content counts don't match. Fixes issues with MGS Master Collection games.
Takes advantage of the recently implemented devoptab interfaces. Supports individual file dumping and multiple file/dir dumping via highlighting. May possibly need some additional testing. The code could potentially be improved, but I'll look into it at a later time -- I just want to release this now.
Other changes include:
* devoptab: comment out log messages.
* utils: add utilsGetDirectorySize() function.
* romfs: add hash bucket fields to RomFileSystemContext struct.
* romfs: remove current offset fields from RomFileSystemContext struct and all functions that relied on them.
* romfs: add romfsCalculateEntryHash().
* romfs: update romfsInitializeContext() to also load file/directory hash buckets.
* romfs: update romfsGetTotalDataSize() to make up for the removed functions.
* romfs: update romfsGetChild*EntryByName() functions to use hash bucket lookups.
Other changes include:
* devoptab: add missing exit macros to some FS operations across all interfaces.
* poc: update extractedRomFsReadThreadFunc() to make up for the removed RomFS functions.
This commit implements a devoptab wrapper for Hash FS sections within gamecard images, using code from hfs.c/h.
Other changes include:
* poc: disable buffering on FILE objects via setvbuf().
* gamecard: fix crash in gamecardInitializeHashFileSystemContext() while calculating the size for the root Hash FS partition.
This commit implements a devoptab wrapper for Partition FS sections within NCAs, using code from pfs.c/h.
Other changes include:
* codebase: use NX_IGNORE_ARG macro where needed.
* hfs: slight tweaks to some of the static functions.
* pfs: slight tweaks to some of the static functions.
* pfs: use pfsIsValidContext() where needed.
* utils: update utilsInitializeResources() to use __system_argc and __system_argv variables from libnx.
* todo: update text file.
