Now using loops instead of recursion. Fixes a nasty stack exhaustion issue while dumping Them's Fightin' Herds raw RomFS section (Compressed+Sparse base with Compressed patch).
Other changes include:
* nca: add title_id and title_type fields to NcaContext; redefine title_version field in NcaContext; redefine ncaInitializeContext() function signature (now requiring NcmContentMetaKey and NcmContentInfo objects); modify ncaInitializeFsSectionContext() to make it skip extra boundary checks on FS sections with a bogus sparse storage.
* config: rename "nsp/write_section_image" flag to "nsp/write_raw_section".
* nxdt_utils: modify utilsGeneratePath() to make it also check if the first character from the filename is a slash.
* pfs: modify pfsGetEntryIndexByName() to only log an error if the entry name being looked up isn't "main.npdm".
Other changes include:
* nca: add content_size_str field to NcaContext struct.
* nxdt_utils: modify utilsGenerateFormattedSizeString() to avoid displaying decimal places if dealing with plain bytes.
Long standing issue since 1bc1a05f96.
I feel extra stupid. It must be the date or something, I don't know. At least I'm getting to fix lots of silly issues within the codebase. Emphasis on "silly" -- I wouldn't have liked to rework the hierarchical patch generation algorithm.
Big thanks to @KookiesGH for notifying about the NSP installation issues with Goldleaf on the Discord server.
And I don't need anyone to tell me so, thank you very much.
* PoC: remove gc_dumper and nsp_dumper PoC; create nxdt_rw_poc with all gc_dumper and nsp_dumper capabilities + standalone ticket dumping + raw NCA dumping; use ftruncate() to set output file sizes whenever possible. PoC code is a mess, as always. Expect the features from the rest of the PoCs to be implemented into nxdt_rw_poc soon.
* workflow: temporarily disable borealis build generation; comment out manual installation of up-to-date packages from Leseratte's mirrors because the latest devkitA64 Docker image has them all.
* borealis: update to fix building issues with latest devkitA64.
* bfttf: error out on invalid NCA signatures.
* config: save configuration to the current working directory; parse and validate new "gamecard/write_raw_hfs_partition" flag.
* defines: remove CONFIG_PATH macro; rename CONFIG_FILE_NAME.
* gamecard: rename fs_ctx -> hfs_ctx everywhere; use HFS function calls to retrieve partition names.
* hfs: move GameCardHashFileSystemPartitionType enum from gamecard.h and rename it to HashFileSystemPartitionType; add hfsIsValidContext(); add hfsGetPartitionNameString().
* nca/npdm: update comments to reflect latest HOS version.
* nxdt_bfsar: always generate absolute SD card paths with the device name; error out on an invalid NCA signature.
* nxdt_includes: include dirent.h; refactor Version struct to make it a union of all known *Version structs.
* nxdt_log: don't write session separator if the logfile is empty.
* nxdt_utils: log appletIsGamePlayRecordingSupported() errors; add utilsDeleteDirectoryRecursively().
* rsa: provide clearer function descriptions in header file.
* services: handle usb:ds initialization.
* tik: update tikConvertPersonalizedTicketToCommonTicket() to allow NULL input pointers as raw certificate chain arguments (much needed for standalone ticket dumping).
* title: add titleGetApplicationIdByMetaKey().
* usb: refactor interface (de)initialization code; slightly improve ABI usage (console-side only); redefine ABI version field in StartSession command blocks; upgrade ABI to v1.1.
* FatFs: rename DIR -> FDIR to avoid conflicts with definitions from stdlib's dirent.h.
* gamecard_tab: display package ID from the inserted gamecard; fix displayed version numbers from bundled system updates below 3.0.0.
* todo: add notes about creating devoptab devices for HFS/PFS/RomFS file tree dumping.
* aes: add aes128EcbCrypt() as a one-shot function to perform AES-128-ECB crypto. The rest of the codebase now calls this function whenever suitable.
* fs_ext: add const keyword to IPC input structs wherever suitable.
* key_sources: add hardcoded master key vectors (prod, dev); master KEK sources (Erista, Mariko); master key source; ticket common key source; SMC key type sources; SMC seal key masks; AES key generation source; NCA header KEK source; NCA header key source and NCA KAEK sources. Also fixed the hardcoded gamecard CardInfo key source for dev units (it was previously generated using retail keydata, my bad).
* keys: remove keysGetNcaMainSignatureModulus(); remove keysDecryptNcaKeyAreaEntry(); repurpose keyset struct to only hold keys that can actually be used for the current hardware type; remove KeysGameCardKeyset; remove keysIsXXModulusYYMandatory() helpers; remove keysRetrieveKeysFromProgramMemory(); remove keysDeriveSealedNcaKeyAreaEncryptionKeys(); add keysDeriveMasterKeys() and keysDerivePerGenerationKeys(); rename keysDeriveGameCardKeys() -> keysDeriveGcCardInfoKey(); add small reimplementations of GenerateAesKek, LoadAesKey and GenerateAesKey; add keysLoadAesKeyFromAesKek() and keysGenerateAesKeyFromAesKek() wrappers. Furthermore, master key derivation is now carried out manually using hardcoded key sources and the last known master key, which is loaded from the Lockpick_RCM keys file -- if the last known master key is unavailable, the key derivation algorithm will then fallback to TSEC root key / Mariko KEK based key derivation, depending on the hardware type.
* nca: add hardcoded NCA man signature moduli (prod, dev); merge ncaDecryptKeyArea() and ncaEncryptKeyArea() into ncaKeyAreaCrypt().
* nxdt_utils: add utilsIsMarikoUnit(); remove _utilsAppletModeCheck(); rename utilsAppletModeCheck() -> utilsIsAppletMode().
* services: remove spl:mig dependency (yay).
* smc: add SmcKeyType enum; add SmcSealKey enum; add SmcGenerateAesKekOption struct; add smcPrepareGenerateAesKekOption().
* Update libusbhsfs to 8c2e517. Furthermore, the UMS handler has been rewritten to use the new callback-based system from libusbhsfs.
* Update FatFs to R0.15 w/patch2.
* gc_dumper: add UMS device support.
* nsp_dumper: add DLC Update support.
* cnmt: add a reminder about the extended data size in NcmContentMetaType_DataPatch CNMTs.
* nca: update NcaKeyGeneration enum comment.
* title: update system titles array, severely overhaul the way linked lists work in Title* structs to properly support DLC updates.
This commit uses my yet unmerged libnx PR to update ncm_types.h.
PoC code hasn't been updated yet, so proper support for DLC updates will arrive at a later time.
Note to self: implement a way to provide access to loaded DataPatch TitleInfo entries (linked list hell).
* bktr: renamed bktrBucketInitializeSubStorageReadParams to bktrInitializeSubStorageReadParams to avoid redundancy, added debug code to dump BucketInfo and BucketTree tables if BucketTree storage initialization fails.
* cnmt: updated ContentMetaAddOnContentMetaExtendedHeader struct to its 15.x equivalent, added ContentMetaLegacyAddOnContentMetaExtendedHeader struct, added ContentMetaDataPatchMetaExtendedHeader struct, updated the cnmtGetRequiredTitleId and cnmtGetRequiredTitleVersion functions to support DataPatch titles, updated cnmtInitializeContext to support both the new AddOnContent extended header and DataPatch titles, added debug code to dump the whole CNMT if context initialization fails, updated cnmtGenerateAuthoringToolXml to support DataPatch titles.
* keys: updated block hashes to match 15.x keyset, use case-insensitive comparison while looking for entry names in keysReadKeysFromFile, make sure the eticket_rsa_kek is non-zero before proceeding in keysGetDecryptedEticketRsaDeviceKey.
* nca: updated NcaKeyGeneration enum, added reminder about updating NcaSignatureKeyGeneration if necessary, replaced ncaFsSectionCheckHashRegionAccess with ncaFsSectionCheckPlaintextHashRegionAccess, removed all extents checks on Patch RomFS and sparse sections, updated ncaGetFsSectionTypeName to reflect if a FS section holds a sparse layer or not.
* nca_storage: updated ncaStorageInitializeContext to avoid initializing a compressed storage if a sparse layer is also used (fixes issues with Them's Fightin' Herds), updated ncaStorageSetPatchOriginalSubStorage to enforce the presence of a compressed storage in a patch if the base FS holds a compressed storage.
* npdm: added reminder about updating NpdmSignatureKeyGeneration if necessary, updated NpdmFsAccessControlFlags enum, updated NpdmAccessibility enum, updated NpdmSystemCallId enum, fixed typos.
* title: updated all relevant functions that deal with NcmContentMetaType values to also handle DataPatch titles, added functions to handle DataPatchId values, removed titleConvertNcmContentSizeToU64 and titleConvertU64ToNcmContentSize functions in favor of ncmContentInfoSizeToU64 and ncmU64ToContentInfoSize from my unmerged libnx PR, updated internal arrays to match 15.x changes, renamed titleOrphanTitleInfoSortFunction to titleInfoEntrySortFunction and updated it to also sort entries by version and storage ID, updated titleGenerateTitleInfoEntriesForTitleStorage to sort TitleInfo entries, simplified titleDuplicateTitleInfo a bit by using macros.
* Use forward declarations for type definitions wherever suitable (e.g. NcaContext, BucketTreeContext).
* Replace references to "DumpDestination" with "OutputStorage".
* Replace references to "append key area" with "prepend key area".
* Update LZ4 to v1.9.4.
* Update Material Icons font to 2022-08-01.
* RootView: change USB-related icons, reflect how many UMS devices are currently connected to the console, provide a way for child views to retrieve the populated UMS devices vector.
* Tasks: implement GetUmsDevices().
* Update borealis.
* Update libusbhsfs.
* Begin implementation of a DumpOptionsFrame class to display configurable options before starting a new dump process.
* bktr: rename bktrIsValidSubstorage() to bktrIsValidSubStorage(), use bktrIsValidSubStorage() in bktrInitializeCompressedStorageContext(), fixed a bug in bktrReadCompressedStorage() where Sparse substorages wouldn't be allowed.
* GamecardTab: push a DumpOptionsFrame object onto the view stack if the XCI option is clicked.
* Update libusbhsfs.
* Update borealis.
* nsp_dumper: force free size retrieval after dumping an NSP.
* title: add titleGetGameCardApplicationMetadataEntries().
* Makefile: remove -gdwarf-4 flag.
* nxdt_utils: treat NT_MAX_FILENAME_LENGTH as bytes instead of codepoints, add "TiB" to the array of supported size suffixes.
* GameCardTab: add ProcessGameCardStatus() and PopulateList(), manage list updates in the same fashion as TitlesTab, display message about how to mitigate launch errors after exiting the application, display available applications in the inserted gamecard, display message about how to perform individual operations on the gamecard titles.
* main: add a try/catch block to intercept any possible exceptions thrown while the application is running + use brls::Application::crash() to gracefully exit afterwards. Temporarily disable Applet Mode support.
* exception_handler: use LOG_LEVEL_ERROR.
* LayeredErrorFrame: add GetListFirstFocusableChild().
* title: implemented titleGetNcmStorageIdName().
* nxdt_log: implemented log verbosity levels (debug, info, warning, error, none) and helper macros for each level. The rest of the codebase still needs to be updated to take advantange of this change.
* nxdt_log: implemented auxiliary logging via nxlink, if available.
* nxdt_utils: system CPU/MEM overclocking is now only applied through utilsSetLongRunningProcessState(), as it should have been from the beginning.
* nxdt_utils: nxlink initialization is now carried out without redirecting stdout and/or stderr, entirely removing the need for utilsRestoreConsoleOutput(). utilsGetNxLinkFileDescriptor() is used to send data to the nxlink host via dprintf() in log functions.
Turns out, if we have a compression layer in patches, we can just ignore the one from the base NCA.
bktrIsBlockWithinIndirectStorageRange() still needs to be updated.
ncaStorageSetPatchOriginalSubStorage: allow Patch title version to be equal to the Base title version (e.g. sparse titles).
ncaStorageGetHashTargetExtents: actually set proper storage extents.
bktrGetTreeNodeEntryIndex: fix index lookup algorithm.
bktrStorageNodeFind: fix binary search.
bktrVisitorMoveNext / bktrVisitorMovePrevious: fix integer overflows.
