1
0
Fork 0
mirror of https://github.com/Ryujinx/Ryujinx.git synced 2024-11-14 02:46:39 +00:00
Commit graph

54 commits

Author SHA1 Message Date
gdk
7dd69f2d0e Allocation free tree lookup 2022-09-10 16:23:49 +02:00
gdk
c646638680 Update several methods to use GetNode directly and avoid array allocations 2022-09-10 16:23:49 +02:00
gdk
65f2a82b97 Optimize PlaceholderManager.UnreserveRange 2022-09-10 16:23:49 +02:00
gdk
93dd6d525a Fix potential issue with partial unmap
We must also do the unmap operation with the RWLock, otherwise faults on the unmapped region will cause crashes and the whole thing becomes pointless
2022-09-10 16:23:49 +02:00
gdk
96d4ad952c Fix reprotection regression 2022-09-10 16:23:49 +02:00
gdk
45e520a27c Rewrite PlaceholderManager4KB to use intrusive RBTree, and to coalesce free placeholders
Also make the other placeholder manager use intrusive RBTree, allows the IntervalTree that was added just for this to be deleted
2022-09-10 16:23:49 +02:00
gdkchan
6922862db8
Optimize kernel memory block lookup and consolidate RBTree implementations (#3410)
* Implement intrusive red-black tree, use it for HLE kernel block manager

* Implement TreeDictionary using IntrusiveRedBlackTree

* Implement IntervalTree using IntrusiveRedBlackTree

* Implement IntervalTree (on Ryujinx.Memory) using IntrusiveRedBlackTree

* Make PredecessorOf and SuccessorOf internal, expose Predecessor and Successor properties on the node itself

* Allocation free tree node lookup
2022-08-26 18:21:48 +00:00
Nicholas Rodine
951700fdd8
Removed unused usings. (#3593)
* Removed unused usings.

* Added back using, now that it's used.

* Removed extra whitespace.
2022-08-18 18:04:54 +02:00
riperiperi
14ce9e1567
Move partial unmap handler to the native signal handler (#3437)
* Initial commit with a lot of testing stuff.

* Partial Unmap Cleanup Part 1

* Fix some minor issues, hopefully windows tests.

* Disable partial unmap tests on macos for now

Weird issue.

* Goodbye magic number

* Add COMPlus_EnableAlternateStackCheck for tests

`COMPlus_EnableAlternateStackCheck` is needed for NullReferenceException handling to work on linux after registering the signal handler, due to how dotnet registers its own signal handler.

* Address some feedback

* Force retry when memory is mapped in memory tracking

This case existed before, but returning `false` no longer retries, so it would crash immediately after unprotecting the memory... Now, we return `true` to deliberately retry.

This case existed before (was just broken by this change) and I don't really want to look into fixing the issue right now. Technically, this means that on guest code partial unmaps will retry _due to this_ rather than hitting the handler. I don't expect this to cause any issues.

This should fix random crashes in Xenoblade Chronicles 2.

* Use IsRangeMapped

* Suppress MockMemoryManager.UnmapEvent warning

This event is not signalled by the mock memory manager.

* Remove 4kb mapping
2022-07-29 19:16:29 -03:00
gdkchan
232b1012b0
Fix ThreadingLock deadlock on invalid access and TerminateProcess (#3407) 2022-06-24 02:53:16 +02:00
gdkchan
dd8f97ab9e
Remove freed memory range from tree on memory block disposal (#3347)
* Remove freed memory range from tree on memory block disposal

* PR feedback
2022-06-05 15:12:42 -03:00
gdkchan
54deded929
Fix shared memory leak on Windows (#3319)
* Fix shared memory leak on Windows

* Fix memory leak caused by RO session disposal not decrementing the memory manager ref count

* Fix UnmapViewInternal deadlock

* Was not supposed to add those back
2022-05-05 14:58:59 -03:00
gdkchan
074190e03c
Remove AddProtection count > 0 assert (#3315) 2022-05-04 14:07:10 -03:00
gdkchan
95017b8c66
Support memory aliasing (#2954)
* Back to the origins: Make memory manager take guest PA rather than host address once again

* Direct mapping with alias support on Windows

* Fixes and remove more of the emulated shared memory

* Linux support

* Make shared and transfer memory not depend on SharedMemoryStorage

* More efficient view mapping on Windows (no more restricted to 4KB pages at a time)

* Handle potential access violations caused by partial unmap

* Implement host mapping using shared memory on Linux

* Add new GetPhysicalAddressChecked method, used to ensure the virtual address is mapped before address translation

Also align GetRef behaviour with software memory manager

* We don't need a mirrorable memory block for software memory manager mode

* Disable memory aliasing tests while we don't have shared memory support on Mac

* Shared memory & SIGBUS handler for macOS

* Fix typo + nits + re-enable memory tests

* Set MAP_JIT_DARWIN on x86 Mac too

* Add back the address space mirror

* Only set MAP_JIT_DARWIN if we are mapping as executable

* Disable aliasing tests again (still fails on Mac)

* Fix UnmapView4KB (by not casting size to int)

* Use ref counting on memory blocks to delay closing the shared memory handle until all blocks using it are disposed

* Address PR feedback

* Make RO hold a reference to the guest process memory manager to avoid early disposal

Co-authored-by: nastys <nastys@users.noreply.github.com>
2022-05-02 20:30:02 -03:00
riperiperi
4a892fbdc9
Fix flush action from multiple threads regression (#3311)
If two or more threads encounter a region of memory where a read action has been registered, then they must _both_ wait on the data.

Clearing the action before it completed was causing the null check above to fail, so the action would only be run on the first thread, and the second would end up continuing without waiting. Depending on what the game does, this could be disasterous.

This fixes a regression introduced by #3302 with Pokemon Legends Arceus, and possibly Catherine. There are likely other affected games. What is fixed in that PR should still be fixed.
2022-05-02 12:31:53 +02:00
riperiperi
d64594ec74
Fix various issues with texture sync (#3302)
* Fix various issues with texture sync

A variable called _actionRegistered is used to keep track of whether a tracking action has been registered for a given texture group handle. This variable is set when the action is registered, and should be unset when it is consumed. This is used to skip registering the tracking action if it's already registered, saving some time for render targets that are modified very often.

There were two issues with this. The worst issue was that the tracking action handler exits early if the handle's modified flag is false... which means that it never reset _actionRegistered, as that was done within the Sync() method called later. The second issue was that this variable was set true after the sync action was registered, so it was technically possible for the action to run immediately, set the flag to false, then set it to true.

Both situations would lead to the action never being registered again, as the texture group handle would be sure the action is already registered. This breaks the texture for the remaining runtime, or until it is disposed.

It was also possible for a texture to register sync once, then on future frames the last modified sync number did not update. This may have caused some more minor issues.

Seems to fix the Xenoblade flashing bug. Obviously this needs a lot of testing, since it was random chance. I typically had the most luck getting it to happen by switching time of day on the event theatre screen for a while, then entering the equipment screen by pressing X on an event.

May also fix weird things like random chance air swimming in BOTW, maybe a few texture streaming bugs.

* Exchange rather than CompareExchange
2022-04-29 18:34:11 -03:00
gdkchan
0a24aa6af2
Allow textures to have their data partially mapped (#2629)
* Allow textures to have their data partially mapped

* Explicitly check for invalid memory ranges on the MultiRangeList

* Update GetWritableRegion to also support unmapped ranges
2022-02-22 13:34:16 -03:00
riperiperi
cda659955c
Texture Sync, incompatible overlap handling, data flush improvements. (#2971)
* Initial test for texture sync

* WIP new texture flushing setup

* Improve rules for incompatible overlaps

Fixes a lot of issues with Unreal Engine games. Still a few minor issues (some caused by dma fast path?) Needs docs and cleanup.

* Cleanup, improvements

Improve rules for fast DMA

* Small tweak to group together flushes of overlapping handles.

* Fixes, flush overlapping texture data for ASTC and BC4/5 compressed textures.

Fixes the new Life is Strange game.

* Flush overlaps before init data, fix 3d texture size/overlap stuff

* Fix 3D Textures, faster single layer flush

Note: nosy people can no longer merge this with Vulkan. (unless they are nosy enough to implement the new backend methods)

* Remove unused method

* Minor cleanup

* More cleanup

* Use the More Fun and Hopefully No Driver Bugs method for getting compressed tex too

This one's for metro

* Address feedback, ASTC+ETC to FormatClass

* Change offset to use Span slice rather than IntPtr Add

* Fix this too
2022-01-09 13:28:48 -03:00
Mary
00c69f2098
Remove usage of Mono.Posix.NETStandard accross all projects (#2906)
* Remove usage of Mono.Posix.NETStandard in Ryujinx project

* Remove usage of Mono.Posix.NETStandard in ARMeilleure project

* Remove usage of Mono.Posix.NETStandard in Ryujinx.Memory project

* Address gdkchan's comments
2021-12-08 18:24:26 -03:00
Mary
f39fce8f54
misc: Migrate usage of RuntimeInformation to OperatingSystem (#2901)
Very basic migration across the codebase.
2021-12-04 20:02:30 -03:00
Mary
57d3296ba4
infra: Migrate to .NET 6 (#2829)
* infra: Migrate to .NET 6

* Rollback version naming change

* Workaround .NET 6 ZipArchive API issues

* ci: Switch to VS 2022 for AppVeyor

CI is now ready for .NET 6

* Suppress WebClient warning in DoUpdateWithMultipleThreads

* Attempt to workaround System.Drawing.Common changes on 6.0.0

* Change keyboard rendering from System.Drawing to ImageSharp

* Make the software keyboard renderer multithreaded

* Bump ImageSharp version to 1.0.4 to fix a bug in Image.Load

* Add fallback fonts to the keyboard renderer

* Fix warnings

* Address caian's comment

* Clean up linux workaround as it's uneeded now

* Update readme

Co-authored-by: Caian Benedicto <caianbene@gmail.com>
2021-11-28 21:24:17 +01:00
Mary
b4dc33efc2
kernel: Clear pages allocated with SetHeapSize (#2776)
* kernel: Clear pages allocated with SetHeapSize

Before this commit, all new pages allocated by SetHeapSize were not
cleared by the kernel.

This would cause undefined data to be pass to the userland and possibly
resulting in weird memory corruption.

This commit also add support for custom fill heap and ipc value (that is also
supported by the official kernel)

* Remove dots at the end of KPageTableBase.MapPages new documentation

* Remove unused _stackFillValue
2021-10-24 18:52:59 -03:00
Mary
85d8d1d7ca
misc: Fix IVirtualMemoryManager.Fill ignoring value (#2775)
This fix IVirtualMemoryManager.Fill to actually use the provided fill
value instead of 0.

This have no implication at the moment as everything that use it pass 0
but it is needed for some upcoming kernel fixes.
2021-10-24 18:16:59 -03:00
riperiperi
fff48bb45a
Smaller initial size for ModifiedRangeList & directly inherit range list (#2663)
This fixes a potential regression with the new range list changes, where the cost for creating new ones would be rather large due to creating a 1024 size array. Also reduces cost for range list inheritance by using the first existing range list as a base, rather than creating a new one then adding both lists to it.

The growth size for the RangeList is now identical to its initial size. Every 32 elements was probably a little too common - now it is 1024 for most things and 8 for the buffer modified range list.

The Unmapped and SyncMethod methods have been changed to ensure that they behave properly if the range list is set null. Cleaned up a few calls to use the null-conditional operator.
2021-10-04 15:38:59 -03:00
riperiperi
d92fff541b
Replace CacheResourceWrite with more general "precise" write (#2684)
* Replace CacheResourceWrite with more general "precise" write

The goal of CacheResourceWrite was to notify GPU resources when they were modified directly, by looking up the modified address/size in a structure and calling a method on each resource. The downside of this is that each resource cache has to be queried individually, they all have to implement their own way to do this, and it can only signal to resources using the same PhysicalMemory instance.

This PR adds the ability to signal a write as "precise" on the tracking, which signals a special handler (if present) which can be used to avoid unnecessary flush actions, or maybe even more. For buffers, precise writes specifically do not flush, and instead punch a hole in the modified range list to indicate that the data on GPU has been replaced.

The downside is that precise actions must ignore the page protection bits and always signal - as they need to notify the target resource to ignore the sequence number optimization.

I had to reintroduce the sequence number increment after I2M, as removing it was causing issues in rabbids kingdom battle. However - all resources modified by I2M are notified directly to lower their sequence number, so the problem is likely that another unrelated resource is not being properly updated. Thankfully, doing this does not affect performance in the games I tested.

This should fix regressions from #2624. Test any games that were broken by that. (RF4, rabbids kingdom battle)

I've also added a sequence number increment to ThreedClass.IncrementSyncpoint, as it seems to fix buffer corruption in OpenGL homebrew. (this was a regression from removing sequence number increment from constant buffer update - another unrelated resource thing)

* Add tests.

* Add XML docs for GpuRegionHandle

* Skip UpdateProtection if only precise actions were called

This allows precise actions to skip reprotection costs.
2021-09-29 02:27:03 +02:00
riperiperi
db97b1d7d2
Implement and use an Interval Tree for the MultiRangeList (#2641)
* Implement and use an Interval Tree for the MultiRangeList

* Feedback

* Address Feedback

* Missed this somehow
2021-09-19 14:55:07 +02:00
riperiperi
7379bc2f39
Array based RangeList that caches Address/EndAddress (#2642)
* Array based RangeList that caches Address/EndAddress

In isolation, this was more than 2x faster than the RangeList that checks using the interface. In practice I'm seeing much better results than I expected. The array is used because checking it is slightly faster than using a list, which loses time to struct copies, but I still want that data locality.

A method has been added to the list to update the cached end address, as some users of the RangeList currently modify it dynamically.

Greatly improves performance in Super Mario Odyssey, Xenoblade and any other GPU limited games.

* Address Feedback
2021-09-19 14:22:26 +02:00
riperiperi
54adc5f9fb
Ensure that all threads wait for a read tracking action to complete. (#2597)
* Lock around tracking action consume + execute. Not particularly fast.

* Lock around preaction registration and use

* Create a lock object

* Nit
2021-08-29 16:03:41 -03:00
riperiperi
ec3e848d79
Add a Multithreading layer for the GAL, multi-thread shader compilation at runtime (#2501)
* Initial Implementation

About as fast as nvidia GL multithreading, can be improved with faster command queuing.

* Struct based command list

Speeds up a bit. Still a lot of time lost to resource copy.

* Do shader init while the render thread is active.

* Introduce circular span pool V1

Ideally should be able to use structs instead of references for storing these spans on commands. Will try that next.

* Refactor SpanRef some more

Use a struct to represent SpanRef, rather than a reference.

* Flush buffers on background thread

* Use a span for UpdateRenderScale.

Much faster than copying the array.

* Calculate command size using reflection

* WIP parallel shaders

* Some minor optimisation

* Only 2 max refs per command now.

The command with 3 refs is gone. 😌

* Don't cast on the GPU side

* Remove redundant casts, force sync on window present

* Fix Shader Cache

* Fix host shader save.

* Fixup to work with new renderer stuff

* Make command Run static, use array of delegates as lookup

Profile says this takes less time than the previous way.

* Bring up to date

* Add settings toggle. Fix Muiltithreading Off mode.

* Fix warning.

* Release tracking lock for flushes

* Fix Conditional Render fast path with threaded gal

* Make handle iteration safe when releasing the lock

This is mostly temporary.

* Attempt to set backend threading on driver

Only really works on nvidia before launching a game.

* Fix race condition with BufferModifiedRangeList, exceptions in tracking actions

* Update buffer set commands

* Some cleanup

* Only use stutter workaround when using opengl renderer non-threaded

* Add host-conditional reservation of counter events

There has always been the possibility that conditional rendering could use a query object just as it is disposed by the counter queue. This change makes it so that when the host decides to use host conditional rendering, the query object is reserved so that it cannot be deleted. Counter events can optionally start reserved, as the threaded implementation can reserve them before the backend creates them, and there would otherwise be a short amount of time where the counter queue could dispose the event before a call to reserve it could be made.

* Address Feedback

* Make counter flush tracked again.

Hopefully does not cause any issues this time.

* Wait for FlushTo on the main queue thread.

Currently assumes only one thread will want to FlushTo (in this case, the GPU thread)

* Add SDL2 headless integration

* Add HLE macro commands.

Co-authored-by: Mary <mary@mary.zone>
2021-08-27 00:31:29 +02:00
gdkchan
bb8a920b63
Do not dirty memory tracking region handles if they are partially unmapped (#2536) 2021-08-11 21:50:33 +02:00
riperiperi
4b60371e64
Return mapped buffer pointer directly for flush, WriteableRegion for textures (#2494)
* Return mapped buffer pointer directly for flush, WriteableRegion for textures

A few changes here to generally improve performance, even for platforms not using the persistent buffer flush.

- Texture and buffer flush now return a ReadOnlySpan<byte>. It's guaranteed that this span is pinned in memory, but it will be overwritten on the next flush from that thread, so it is expected that the data is used before calling again.
- As a result, persistent mappings no longer copy to a new array - rather the persistent map is returned directly as a Span<>. A similar host array is used for the glGet flushes instead of allocating new arrays each time.
- Texture flushes now do their layout conversion into a WriteableRegion when the texture is not MultiRange, which allows the flush to happen directly into guest memory rather than into a temporary span, then copied over. This avoids another copy when doing layout conversion.

Overall, this saves 1 data copy for buffer flush, 1 copy for linear textures with matching source/target stride, and 2 copies for block textures or linear textures with mismatching strides.

* Fix tests

* Fix array pointer for Mesa/Intel path

* Address some feedback

* Update method for getting array pointer.
2021-07-19 19:10:54 -03:00
gdkchan
b1a9d17cf8
Fix GetWritableRegion write-back (#2456) 2021-07-10 21:50:10 +02:00
riperiperi
12a7a2ead8
Inherit buffer tracking handles rather than recreating on resize (#2330)
This greatly speeds up games that constantly resize buffers, and removes stuttering on games that resize large buffers occasionally:

- Large improvement on Super Mario 3D All-Stars (#1663 needed for best performance)
- Improvement to Hyrule Warriors: AoC, and UE4 games. These games can still stutter due to texture creation/loading.
- Small improvement to other games, potential 1-frame stutters avoided.

`ForceSynchronizeMemory`, which was added with POWER, is no longer needed. Some tests have been added for the MultiRegionHandle.
2021-06-24 01:31:26 +02:00
riperiperi
54ea2285f0
POWER - Performance Optimizations With Extensive Ramifications (#2286)
* Refactoring of KMemoryManager class

* Replace some trivial uses of DRAM address with VA

* Get rid of GetDramAddressFromVa

* Abstracting more operations on derived page table class

* Run auto-format on KPageTableBase

* Managed to make TryConvertVaToPa private, few uses remains now

* Implement guest physical pages ref counting, remove manual freeing

* Make DoMmuOperation private and call new abstract methods only from the base class

* Pass pages count rather than size on Map/UnmapMemory

* Change memory managers to take host pointers

* Fix a guest memory leak and simplify KPageTable

* Expose new methods for host range query and mapping

* Some refactoring of MapPagesFromClientProcess to allow proper page ref counting and mapping without KPageLists

* Remove more uses of AddVaRangeToPageList, now only one remains (shared memory page checking)

* Add a SharedMemoryStorage class, will be useful for host mapping

* Sayonara AddVaRangeToPageList, you served us well

* Start to implement host memory mapping (WIP)

* Support memory tracking through host exception handling

* Fix some access violations from HLE service guest memory access and CPU

* Fix memory tracking

* Fix mapping list bugs, including a race and a error adding mapping ranges

* Simple page table for memory tracking

* Simple "volatile" region handle mode

* Update UBOs directly (experimental, rough)

* Fix the overlap check

* Only set non-modified buffers as volatile

* Fix some memory tracking issues

* Fix possible race in MapBufferFromClientProcess (block list updates were not locked)

* Write uniform update to memory immediately, only defer the buffer set.

* Fix some memory tracking issues

* Pass correct pages count on shared memory unmap

* Armeilleure Signal Handler v1 + Unix changes

Unix currently behaves like windows, rather than remapping physical

* Actually check if the host platform is unix

* Fix decommit on linux.

* Implement windows 10 placeholder shared memory, fix a buffer issue.

* Make PTC version something that will never match with master

* Remove testing variable for block count

* Add reference count for memory manager, fix dispose

Can still deadlock with OpenAL

* Add address validation, use page table for mapped check, add docs

Might clean up the page table traversing routines.

* Implement batched mapping/tracking.

* Move documentation, fix tests.

* Cleanup uniform buffer update stuff.

* Remove unnecessary assignment.

* Add unsafe host mapped memory switch

On by default. Would be good to turn this off for untrusted code (homebrew, exefs mods) and give the user the option to turn it on manually, though that requires some UI work.

* Remove C# exception handlers

They have issues due to current .NET limitations, so the meilleure one fully replaces them for now.

* Fix MapPhysicalMemory on the software MemoryManager.

* Null check for GetHostAddress, docs

* Add configuration for setting memory manager mode (not in UI yet)

* Add config to UI

* Fix type mismatch on Unix signal handler code emit

* Fix 6GB DRAM mode.

The size can be greater than `uint.MaxValue` when the DRAM is >4GB.

* Address some feedback.

* More detailed error if backing memory cannot be mapped.

* SetLastError on all OS functions for consistency

* Force pages dirty with UBO update instead of setting them directly.

Seems to be much faster across a few games. Need retesting.

* Rebase, configuration rework, fix mem tracking regression

* Fix race in FreePages

* Set memory managers null after decrementing ref count

* Remove readonly keyword, as this is now modified.

* Use a local variable for the signal handler rather than a register.

* Fix bug with buffer resize, and index/uniform buffer binding.

Should fix flickering in games.

* Add InvalidAccessHandler to MemoryTracking

Doesn't do anything yet

* Call invalid access handler on unmapped read/write.

Same rules as the regular memory manager.

* Make unsafe mapped memory its own MemoryManagerType

* Move FlushUboDirty into UpdateState.

* Buffer dirty cache, rather than ubo cache

Much cleaner, may be reusable for Inline2Memory updates.

* This doesn't return anything anymore.

* Add sigaction remove methods, correct a few function signatures.

* Return empty list of physical regions for size 0.

* Also on AddressSpaceManager

Co-authored-by: gdkchan <gab.dark.100@gmail.com>
2021-05-24 22:52:44 +02:00
riperiperi
a539303e71
Remove unused physical region tracking (#2085)
* Remove unused physical region tracking

* Update comments
2021-03-06 20:21:53 -03:00
riperiperi
8d36681bf1
Improve handling for unmapped GPU resources (#2083)
* Improve handling for unmapped GPU resources

- Fixed a memory tracking bug that would set protection on empty PTEs
- When a texture's memory is (partially) unmapped, all pool references are forcibly removed and the texture must be rediscovered to draw with it. This will also force the texture discovery to always compare the texture's range for a match.
- RegionHandles now know if they are unmapped, and automatically unset their dirty flag when unmapped.
- Partial texture sync now loads only the region of texture that has been modified. Unmapped memory tracking handles cause dirty flags for a texture group handle to be ignored.

This greatly improves the emulator's stability for newer UE4 games.

* Address feedback, fix MultiRange slice

Fixed an issue where the size of the multi-range slice would be miscalculated.

* Update Ryujinx.Memory/Range/MultiRange.cs (feedback)

Co-authored-by: Mary <thog@protonmail.com>

Co-authored-by: Mary <thog@protonmail.com>
2021-03-06 11:43:55 -03:00
riperiperi
b530f0e110
Texture Cache: "Texture Groups" and "Texture Dependencies" (#2001)
* Initial implementation (3d tex mips broken)

This works rather well for most games, just need to fix 3d texture mips.

* Cleanup

* Address feedback

* Copy Dependencies and various other fixes

* Fix layer/level offset for copy from view<->view.

* Remove dirty flag from dependency

The dirty flag behaviour is not needed - DeferredCopy is all we need.

* Fix tracking mip slices.

* Propagate granularity (fix astral chain)

* Address Feedback pt 1

* Save slice sizes as part of SizeInfo

* Fix nits

* Fix disposing multiple dependencies causing a crash

This list is obviously modified when removing dependencies, so create a copy of it.
2021-03-02 19:30:54 -03:00
Caian Benedicto
f7b2daf5ec
Fix virtual address overflow near ulong limit (#2044)
* Fix virtual address overflow near ulong limit

* Fix comments

* Improve overflow checking for large size values

* Add overflow checking to AddressSpaceManager class

* Add overflow protection to read and write functions
2021-02-28 11:48:04 +01:00
LDj3SNuD
dc0adb533d
PPTC & Pool Enhancements. (#1968)
* PPTC & Pool Enhancements.

* Avoid buffer allocations in CodeGenContext.GetCode(). Avoid stream allocations in PTC.PtcInfo.

Refactoring/nits.

* Use XXHash128, for Ptc.Load & Ptc.Save, x10 faster than Md5.

* Why not a nice Span.

* Added a simple PtcFormatter library for deserialization/serialization, which does not require reflection, in use at PtcJumpTable and PtcProfiler; improves maintainability and simplicity/readability of affected code.

* Nits.

* Revert #1987.

* Revert "Revert #1987."

This reverts commit 998be765cf.
2021-02-22 03:23:48 +01:00
gdkchan
734747ae58
Reduce temporary copy/fill buffer size (#1926) 2021-01-19 13:31:15 +11:00
riperiperi
a1f77a5b6a
Implement lazy flush-on-read for Buffers (SSBO/Copy) (#1790)
* Initial implementation of buffer flush (VERY WIP)

* Host shaders need to be rebuilt for the SSBO write flag.

* New approach with reserved regions and gl sync

* Fix a ton of buffer issues.

* Remove unused buffer unmapped behaviour

* Revert "Remove unused buffer unmapped behaviour"

This reverts commit f1700e52fb8760180ac5e0987a07d409d1e70ece.

* Delete modified ranges on unmap

Fixes potential crashes in Super Smash Bros, where a previously modified range could lie on either side of an unmap.

* Cache some more delegates.

* Dispose Sync on Close

* Also create host sync for GPFifo syncpoint increment.

* Copy buffer optimization, add docs

* Fix race condition with OpenGL Sync

* Enable read tracking on CommandBuffer, insert syncpoint on WaitForIdle

* Performance: Only flush individual pages of SSBO at a time

This avoids flushing large amounts of data when only a small amount is actually used.

* Signal Modified rather than flushing after clear

* Fix some docs and code style.

* Introduce a new test for tracking memory protection.

Sucessfully demonstrates that the bug causing write protection to be cleared by a read action has been fixed. (these tests fail on master)

* Address Comments

* Add host sync for SetReference

This ensures that any indirect draws will correctly flush any related buffer data written before them. Fixes some flashing and misplaced world geometry in MH rise.

* Make PageAlign static

* Re-enable read tracking, for reads.
2021-01-17 17:08:06 -03:00
gdkchan
c4f56c5704
Support for resources on non-contiguous GPU memory regions (#1905)
* Support for resources on non-contiguous GPU memory regions

* Implement MultiRange physical addresses, only used with a single range for now

* Actually use non-contiguous ranges

* GetPhysicalRegions fixes

* Documentation and remove Address property from TextureInfo

* Finish implementing GetWritableRegion

* Fix typo
2021-01-17 19:44:34 +01:00
LDj3SNuD
68f6b79fd3
Add a simple Pools Limiter. (#1830)
* Added support for offline invalidation, via PPTC, of low cq translations replaced by high cq translations; both on a single run and between runs.

Added invalidation of .cache files in the event of reuse on a different user operating system.

Added .info and .cache files invalidation in case of a failed stream decompression.

Nits.

* InternalVersion = 1712;

* Nits.

* Address comment.

* Get rid of BinaryFormatter.

Nits.

* Move Ptc.LoadTranslations().

Nits.

* Nits.

* Fixed corner cases (in case backup copies have to be used). Added save logs.

* Not core fixes.

* Complement to the previous commit. Added load logs. Removed BinaryFormatter leftovers.

* Add LoadTranslations log.

* Nits.

* Removed the search and management of LowCq overlapping functions.

* Final increment of .info and .cache flags.

* Nit.

* Free up memory allocated by Pools during any PPTC translations at boot time.

* Nit due to rebase.

* Add a simple Pools Limiter.

* Nits.

* Fix missing JumpTable.RegisterFunction() due to rebase.

Clear MemoryStreams as soon as possible, when they are no longer needed.

* Code cleaning.

* Nit for retrigger Checks.

* Update Ptc.cs

* Contextual refactoring of Translator. Ignore resetting of pools for DirectCallStubs.

* Nit for retrigger Checks.
2021-01-12 19:04:02 +01:00
EmulationFanatic
d05db79a3e
Update InvalidMemoryRegionException.cs to abide by English rules. (#1834) 2020-12-25 04:09:02 +01:00
gdkchan
19d18662ea
Correct type of executable sizes (#1802) 2020-12-13 08:30:27 +01:00
riperiperi
4594c3b310
Signal memory tracking before/after mapping into another process (#1785)
* Signal memory tracking before/after mapping into another process

* Wording.

* Add missing method.
2020-12-07 21:42:17 -03:00
gdkchan
cf6cd71488
IPC refactor part 2: Use ReplyAndReceive on HLE services and remove special handling from kernel (#1458)
* IPC refactor part 2: Use ReplyAndReceive on HLE services and remove special handling from kernel

* Fix for applet transfer memory + some nits

* Keep handles if possible to avoid server handle table exhaustion

* Fix IPC ZeroFill bug

* am: Correctly implement CreateManagedDisplayLayer and implement CreateManagedDisplaySeparableLayer

CreateManagedDisplaySeparableLayer is requires since 10.x+ when appletResourceUserId != 0

* Make it exit properly

* Make ServiceNotImplementedException show the full message again

* Allow yielding execution to avoid starving other threads

* Only wait if active

* Merge IVirtualMemoryManager and IAddressSpaceManager

* Fix Ro loading data from the wrong process

Co-authored-by: Thog <me@thog.eu>
2020-12-02 00:23:43 +01:00
Mary
aa129fdbdf
infra: Migrate to .NET 5 (#1694)
* infra: Migrate to .NET 5

This migrate projects and CI to .NET 5

* Remove language version restrictions (now on 9.0 by default)

* infra: pin .NET 5 to avoid later issues

* infra: Cleanup csproj files

* infra: update dependencies

* infra: Add temporary workaround for a bug in Vector128.Create

see https://github.com/dotnet/runtime/issues/44704 for more informations
2020-11-15 19:27:15 +01:00
gdkchan
3c60d4b0ea
Do not report unmapped pages as dirty (#1672)
* Do not report unmapped pages as dirty

* Make tests pass again

* PR feedback
2020-11-10 22:07:52 -03:00
riperiperi
b4d8d893a4
Memory Read/Write Tracking using Region Handles (#1272)
* WIP Range Tracking

- Texture invalidation seems to have large problems
- Buffer/Pool invalidation may have problems
- Mirror memory tracking puts an additional `add` in compiled code, we likely just want to make HLE access slower if this is the final solution.
- Native project is in the messiest possible location.
- [HACK] JIT memory access always uses native "fast" path
- [HACK] Trying some things with texture invalidation and views.

It works :)

Still a few hacks, messy things, slow things

More work in progress stuff (also move to memory project)

Quite a bit faster now.
- Unmapping GPU VA and CPU VA will now correctly update write tracking regions, and invalidate textures for the former.
- The Virtual range list is now non-overlapping like the physical one.
- Fixed some bugs where regions could leak.
- Introduced a weird bug that I still need to track down (consistent invalid buffer in MK8 ribbon road)

Move some stuff.

I think we'll eventually just put the dll and so for this in a nuget package.

Fix rebase.

[WIP] MultiRegionHandle variable size ranges

- Avoid reprotecting regions that change often (needs some tweaking)
- There's still a bug in buffers, somehow.
- Might want different api for minimum granularity

Fix rebase issue

Commit everything needed for software only tracking.

Remove native components.

Remove more native stuff.

Cleanup

Use a separate window for the background context, update opentk. (fixes linux)

Some experimental changes

Should get things working up to scratch - still need to try some things with flush/modification and res scale.

Include address with the region action.

Initial work to make range tracking work

Still a ton of bugs

Fix some issues with the new stuff.

* Fix texture flush instability

There's still some weird behaviour, but it's much improved without this. (textures with cpu modified data were flushing over it)

* Find the destination texture for Buffer->Texture full copy

Greatly improves performance for nvdec videos (with range tracking)

* Further improve texture tracking

* Disable Memory Tracking for view parents

This is a temporary approach to better match behaviour on master (where invalidations would be soaked up by views, rather than trigger twice)

The assumption is that when views are created to a texture, they will cover all of its data anyways. Of course, this can easily be improved in future.

* Introduce some tracking tests.

WIP

* Complete base tests.

* Add more tests for multiregion, fix existing test.

* Cleanup Part 1

* Remove unnecessary code from memory tracking

* Fix some inconsistencies with 3D texture rule.

* Add dispose tests.

* Use a background thread for the background context.

Rather than setting and unsetting a context as current, doing the work on a dedicated thread with signals seems to be a bit faster.

Also nerf the multithreading test a bit.

* Copy to texture with matching alignment

This extends the copy to work for some videos with unusual size, such as tutorial videos in SMO. It will only occur if the destination texture already exists at XCount size.

* Track reads for buffer copies. Synchronize new buffers before copying overlaps.

* Remove old texture flushing mechanisms.

Range tracking all the way, baby.

* Wake the background thread when disposing.

Avoids a deadlock when games are closed.

* Address Feedback 1

* Separate TextureCopy instance for background thread

Also `BackgroundContextWorker.InBackground` for a more sensible idenfifier for if we're in a background thread.

* Add missing XML docs.

* Address Feedback

* Maybe I should start drinking coffee.

* Some more feedback.

* Remove flush warning, Refocus window after making background context
2020-10-16 17:18:35 -03:00