* gc_dumper: add UMS device support.
* nsp_dumper: add DLC Update support.
* cnmt: add a reminder about the extended data size in NcmContentMetaType_DataPatch CNMTs.
* nca: update NcaKeyGeneration enum comment.
* title: update system titles array, severely overhaul the way linked lists work in Title* structs to properly support DLC updates.
This commit uses my yet unmerged libnx PR to update ncm_types.h.
PoC code hasn't been updated yet, so proper support for DLC updates will arrive at a later time.
Note to self: implement a way to provide access to loaded DataPatch TitleInfo entries (linked list hell).
* bktr: renamed bktrBucketInitializeSubStorageReadParams to bktrInitializeSubStorageReadParams to avoid redundancy, added debug code to dump BucketInfo and BucketTree tables if BucketTree storage initialization fails.
* cnmt: updated ContentMetaAddOnContentMetaExtendedHeader struct to its 15.x equivalent, added ContentMetaLegacyAddOnContentMetaExtendedHeader struct, added ContentMetaDataPatchMetaExtendedHeader struct, updated the cnmtGetRequiredTitleId and cnmtGetRequiredTitleVersion functions to support DataPatch titles, updated cnmtInitializeContext to support both the new AddOnContent extended header and DataPatch titles, added debug code to dump the whole CNMT if context initialization fails, updated cnmtGenerateAuthoringToolXml to support DataPatch titles.
* keys: updated block hashes to match 15.x keyset, use case-insensitive comparison while looking for entry names in keysReadKeysFromFile, make sure the eticket_rsa_kek is non-zero before proceeding in keysGetDecryptedEticketRsaDeviceKey.
* nca: updated NcaKeyGeneration enum, added reminder about updating NcaSignatureKeyGeneration if necessary, replaced ncaFsSectionCheckHashRegionAccess with ncaFsSectionCheckPlaintextHashRegionAccess, removed all extents checks on Patch RomFS and sparse sections, updated ncaGetFsSectionTypeName to reflect if a FS section holds a sparse layer or not.
* nca_storage: updated ncaStorageInitializeContext to avoid initializing a compressed storage if a sparse layer is also used (fixes issues with Them's Fightin' Herds), updated ncaStorageSetPatchOriginalSubStorage to enforce the presence of a compressed storage in a patch if the base FS holds a compressed storage.
* npdm: added reminder about updating NpdmSignatureKeyGeneration if necessary, updated NpdmFsAccessControlFlags enum, updated NpdmAccessibility enum, updated NpdmSystemCallId enum, fixed typos.
* title: updated all relevant functions that deal with NcmContentMetaType values to also handle DataPatch titles, added functions to handle DataPatchId values, removed titleConvertNcmContentSizeToU64 and titleConvertU64ToNcmContentSize functions in favor of ncmContentInfoSizeToU64 and ncmU64ToContentInfoSize from my unmerged libnx PR, updated internal arrays to match 15.x changes, renamed titleOrphanTitleInfoSortFunction to titleInfoEntrySortFunction and updated it to also sort entries by version and storage ID, updated titleGenerateTitleInfoEntriesForTitleStorage to sort TitleInfo entries, simplified titleDuplicateTitleInfo a bit by using macros.
* Update libusbhsfs.
* Update borealis.
* nsp_dumper: force free size retrieval after dumping an NSP.
* title: add titleGetGameCardApplicationMetadataEntries().
* Makefile: remove -gdwarf-4 flag.
* nxdt_utils: treat NT_MAX_FILENAME_LENGTH as bytes instead of codepoints, add "TiB" to the array of supported size suffixes.
* GameCardTab: add ProcessGameCardStatus() and PopulateList(), manage list updates in the same fashion as TitlesTab, display message about how to mitigate launch errors after exiting the application, display available applications in the inserted gamecard, display message about how to perform individual operations on the gamecard titles.
* main: add a try/catch block to intercept any possible exceptions thrown while the application is running + use brls::Application::crash() to gracefully exit afterwards. Temporarily disable Applet Mode support.
* exception_handler: use LOG_LEVEL_ERROR.
* LayeredErrorFrame: add GetListFirstFocusableChild().
* title: implemented titleGetNcmStorageIdName().
* nxdt_log: implemented log verbosity levels (debug, info, warning, error, none) and helper macros for each level. The rest of the codebase still needs to be updated to take advantange of this change.
* nxdt_log: implemented auxiliary logging via nxlink, if available.
* nxdt_utils: system CPU/MEM overclocking is now only applied through utilsSetLongRunningProcessState(), as it should have been from the beginning.
* nxdt_utils: nxlink initialization is now carried out without redirecting stdout and/or stderr, entirely removing the need for utilsRestoreConsoleOutput(). utilsGetNxLinkFileDescriptor() is used to send data to the nxlink host via dprintf() in log functions.
ncaStorageSetPatchOriginalSubStorage: allow Patch title version to be equal to the Base title version (e.g. sparse titles).
ncaStorageGetHashTargetExtents: actually set proper storage extents.
bktrGetTreeNodeEntryIndex: fix index lookup algorithm.
bktrStorageNodeFind: fix binary search.
bktrVisitorMoveNext / bktrVisitorMovePrevious: fix integer overflows.
* Check for sparse layers in nsp_dumper and xml_generator PoCs before attempting to initialize content-type-specific contexts.
* system_title_dumper now builds.
* gamecardCloseStorageArea() no longer returns prematurely on application exit.
* Fix section size for sparse sections in ncaInitializeFsSectionContext().
* Fix ncaFsSectionValidateHashDataBoundaries().
* Implement bktrInitializeIndirectStorage() and bktrInitializeAesCtrExStorage().
* gc_dumper: fixed Card ID Set dumping.
* nacp: updated structs and enums to match latest changes (big thanks to @0Liam !), added helper macros.
* cnmt: added helper macros.
* program_info: added helper macros.
* exception_handler: added helper macros.
* nxdt_utils: only create output directories if the app isn't a PoC build.
* todo: changed task priority order. Sparse/compressed section support is mandatory and must be implemented ASAP.
* nxdt_includes: renamed VersionType1 to Version and tweaked it to add the application_version struct, renamed VersionType2 to SdkAddOnVersion.
* gc_dumper: use utilsGeneratePath() to truncate generated filenames to safe filesystem limits. Thanks to n0tw0rk for reporting the issue and testing the changes!
* nxdt_host.py: fixed a string formatting issue that made it impossible to unpack data from the SendFileProperties command block under certain circumstances.
* gamecard: modified GameCardKeySource to add a value field, added GameCardFlags_HasCa10Certificate flag.
* utils: utilsGeneratePath() no longer adds a dot on its own if it's not part of the provided extension string.
Other changes include:
* Codestyle fixes.
* Remove references to GameCardKeyArea in the usb_gc_dumper PoC.
* Remove option to append key area to output XCI dumps in usb_gc_dumper PoC.
* Add support for dumping full GameCardSecurityInformation
* Add support for dumping LAFW
* Clear out ASIC session hash data
Co-authored-by: Pablo Curiel <pabloacurielz@gmail.com>
* Both gamecard header and decrypted CardInfo area are now retrieved upon gamecard insertion. LAFW version is checked against the CardInfo LAFW version right afterwards.
* Expanded GameCardStatus enum to add NoGameCardPatchEnabled and LotusAsicFirmwareUpdateRequired values.
* Updated utilsReplaceIllegalCharacters() to perform replacements on a per-codepoint basis, which means that invalid multibyte UTF-8 codepoints can now be replaced with a single ASCII underscore.
* Updated utilsGeneratePath() to truncate path elements that exceed 255 UTF-8 codepoints (safe limit for FAT and NTFS filesystems).
* Heavily simplified core logic in title functions by using newly defined TitleStorage elements (which hold the NCM database/storage handles, a TitleInfo array and a title counter) instead of the old, global index-based methods.
* Simplified background gamecard title thread logic by always returning duplicated TitleInfo data to the user.
* Update title API to account for the previously mentioned changes, including functions to free duplicated title data.
* Fallback gamecard filename string now also holds the gamecard package ID whenever possible.
* Implemented HDCP patching for Control NCAs.
* Implemented RSA-2048-PSS + SHA256 signature verification.
* Refactored RSA-2048-OAEP decryption steps to use mbedtls function calls.
* Implemented NCA header main signature verification.
* Replaced Björn Samuelsson's CRC32 algorithm with the hardware accelerated CRC32 checksum calculation from libnx (latest commit with support for calculation in blocks).
* Rewrote mutex handling throughout the code to use a small, macro-based scoped lock implementation.
* Removed extern variables from common.h - launch path management is now completely handled in utils.c.
* Updated NpdmSystemCallId_Count to reflect changes introduced in 12.0.0.
* Added NcaMainSignatureKeyGeneration enum.
* NCA main signature moduli are now retrieved from FS .rodata at runtime.
* Simplified lock management in usb.c by using a single global mutex with scoped locks instead of three different r/w locks.
* Updated FatFs to R0.14b.
* Enabled 64-bit LBA support in FatFs to potentially support custom eMMC replacements / resized USER partitions in the future.
* Updated LZ4 to v1.9.3.
* Fixed typos.
* USB gamecard dumper PoC now only dumps the Initial Data area.
* Updated to-do list.
* Refactored keydata handling.
* Sealed NCA KAEKs are now generated at startup, and NCA key area entries are now decrypted by keysDecryptNcaKeyAreaEntry(), reducing the number of calls to spl functions.
* The eTicket RSA device key is now retrieved and decrypted at startup. RSA-OAEP wrapped titlekeys are now decrypted by keysDecryptRsaOaepWrappedTitleKey().
* Renamed titlekek -> ticket common key throughout the codebase.
* Added NcaKeyAreaEncryptionKeyIndex_Count and NcaKeyGeneration_Max enum values to nca.h.
* Proper usage of strcasecmp() in some functions.
* Moved syscall hint checks from keys.c to mem.c.
* Define illegal FS characters as an array rather than a char pointer.
* Services are now initialized before the CFW type checks.
* Fixed pcv/clkrst service initialization.
* Implemented additional thread safety and logfile output to service functions.
* Slightly tweaked running service checks.
* Added proper Markdown documentation for the USB ABI.
* Avoid performing any crypto operations on null NCA key area entries.
* Added commented out code to handle the aes_ctr_ex NCA key area entry, just in case we end up needing it at some point.
* Added functions to deal with title info and application metadata reallocations, greatly reducing the number of references to realloc() throughout the code.
* Tweaked gamecardGetHashFileSystemContext() to not return a pointer to a dynamic context.
* Added a type field to Hash FS contexts.
* usb: copy command ID and block size before moving command data within the USB transfer buffer.
* nsp_dumper_usb: now spans a background thread for the dump process, progress is now displayed, process can now be cancelled by holding B, updated to properly make use of the new usbCancelFileTransfer() behavior.
* usb_gc_dumper: updated to properly make use of the new usbCancelFileTransfer() behavior.
* usb_romfs_dumper: updated to properly make use of the new usbCancelFileTransfer() behavior.
* Updated ns-usbloader patch. Must be used on commit `8771d551a4e6fa2d645e519d504a377e34cbd730`.
* Temporarily changed tracked libusbhsfs branch from main to dev.
* Minor improvements in ums.c/h.
* Refactored nsp_dumper_sd into nsp_dumper_stor - now capable of dumping NSPs to a mounted partition from a UMS device.
* Fixed utilsWaitForButtonPress() when no button mask is provided.
* Refactored utilsGetFreeSpaceFromFileSystemByPath() into utilsGetFileSystemStatsByPath().
* Updated hardcoded system title list to match HOS 11.0.0.
* Updated NcaKeyGeneration_Current.
* Updated button input handling to match the new pad API from libnx.
* Cancel USB file transfer if something goes wrong during a NSP transfer via USB.
* Added SD card RomFS dumper.
* Further optimizations to the *WriteNcaPatch() functions.
* Change struct naming in nso.c/h (thanks @0Liam !).
* Replace fsp-usb with libusbhsfs. ( ͡° ͜ʖ ͡°)
* Fixed possible invalid memory accesses using TitleInfo pointers from gamecard titles if the gamecard they belong to is taken out.
* Fixed invalid for loop condition in _titleGetInfoFromStorageByTitleId() if no gamecard titles have been retrieved.
* Added NSP dumper PoC (SD card only atm, single-threaded).
* Cert: replaced a wrong strcmp() with a proper strncmp().
* CNMT: added functions to update content info entries and generate/write Partition FS patches.
* NCA: encrypt key area right after removing titlekey crypto.
* NPDM/ProgramInfo: changed function names.
* NPDM: check if the NCA has been modified before attempting to patch ACID data + calculate RSA-PSS signature *after* generating the PFS patch, not before. lol
* PFS: restore name table size value before writing the header padding.
* Tik: reworked the ticket lookup algorithm. Now uses information from ticket_list.bin to properly calculate the offset to the requested ticket in ticket.bin.
* Title: changed title type strings used for filename generation.
* Updated to-do list.
* AES: moved CTR initializing/updating functions here from nca.c.
* BKTR/RomFS/PFS: check if we're dealing with a NCA with titlekey crypto and the titlekey hasn't been retrieved.
* BFTTF: use void pointers for output font data.
* Mem: Only exclude Unmapped/Uo/ThreadLocal/Reserved memory pages if dealing with FS.
* NCA: use content type context pointers inside NCA contexts to manage ContentMeta, ProgramInfo, Nacp and LegalInfo contexts.
* NCA: added 'written' bool elements to patch structs to indicate patch write completion.
* NPDM: remove unnecessary inline functions, generate PFS patch right after changing ACID data, add a pfsWriteEntryPatchToMemoryBuffer wrapper.
* PFS: added PartitionFileSystemFileContext and related functions to deal with NSP headers.
* ProgramInfo: removed unnecessary inline functions.
* Save: added commented code to dump a full system savefile - will probably use it down the road.
* Tik: added support for volatile tickets (thanks to @shchmue and @Whovian9369!), added a rights ID string representation to the Ticket struct, clear Volatile and ELicenseRequired flags in conversions to common tickets.
* Title: added a function to calculate the number of titles (current + siblings) from a TItleInfo block.
* Utils: added a function to generate a dynamically allocated path string using a prefix, a filename and a extension.
* Removed explicit offset checks throughout all the code.
* Codestyle fixes.
* Updated to-do.
* Added a function to write re-encrypted NCA and/or FS section headers. Getting ready to re-implement NSP dumping.
* Added more CNMT XML fields (thanks 0Liam).
* Updated my to-do list because I'm dummy.
* Added 'encrypted_header' members to both NcaContext and NcaFsSectionContext. In-place re-encryption isn't desirable in this case.
* Fixed FsAccessControl-related type naming for ACI0 blocks.
Also:
* Fixed support for NSOs with invalid module_name, api_info, dynstr and/or dynsym offsets/sizes.
* Updated the XML generator example to generate XML data for all available NCAs, regardless of their ID offset values.
XML generation confirmed to be working. The new algorithm faithfully reproduces the same output from legacy nxdumptool with much less overhead and memory usage.
From now on, I'll delete code from the legacy codebase directory as soon as specific features from it are rewritten... Because it's a mess to navigate.
Key area and certificate can be optionally appended and/or preserved, respectively. Trimming is supported.
Also added a bunch of ZLT packet related fixes to usb.c, as well as a function to properly cancel an ongoing file transfer.
