Certain types of hooks aren't detected on x64 syscalls.

Created by: lcrobin

The implementation of syscalls_init is insufficient to detect certain types of hooks on x64. Reproduced in 6.2 and 7.0-rc.

An example of the 'normal' syscall stubs on windows 10 is in the comments for that function (syscall index replaced with NNNN to show the issue):

ntdll!ZwApi-PreHook

00007ffc`3ba552d0 4c8bd1           mov     r10,rcx
00007ffc`3ba552d3 b8NNNNNNNN       mov     eax,NNh
00007ffc`3ba552d8 f604250803fe7f01 test    byte ptr [SharedUserData+0x308 (00000000`7ffe0308)],1
00007ffc`3ba552e0 7503             jne     00007ffc3ba552e5
00007ffc`3ba552e2 0f05             syscall
00007ffc`3ba552e4 c3               ret
00007ffc`3ba552e5 cd2e             int     2Eh
00007ffc`3ba552e7 c3               ret

One known implementation of a hook here ignores the first 3 bytes, and instead puts a relative jmp or call in place of the 2nd instruction. This allows the hook and unhook procedure to be completely atomic, as it replaces a single instruction for another instruction. Doing so of course destroys the syscall index that was there before:

ntdll!ZwApi-PostHook

00007ffc`3ba552d0 4c8bd1           mov     r10,rcx
00007ffc`3ba552d3 e9NNNNNNNN       jmp/call +/- NNNNNNNNh
00007ffc`3ba552d8 f604250803fe7f01 test    byte ptr [SharedUserData+0x308 (00000000`7ffe0308)],1
00007ffc`3ba552e0 7503             jne     00007ffc3ba552e5
00007ffc`3ba552e2 0f05             syscall
00007ffc`3ba552e4 c3               ret
00007ffc`3ba552e5 cd2e             int     2Eh
00007ffc`3ba552e7 c3               ret

This is usually accomplished with an additional trampoline instruction somewhere inside of ntdll so the jmp/call destination is within the +/- 2GB supported by this 5 byte instruction.

If this hook happens before dynamorio injects into the child process and does its initialization, the tool will make an incorrect determination on the hook and syscall index.

syscalls_init calls syscalls_init_get_num for two APIs, either of which could be hooked as above. If they are hooked as above, this API will extract some random value for the syscall index because of this code:

    if (wrapper != NULL && !ALLOW_HOOKER(wrapper))
        return *((int *)((wrapper) + SYSNUM_OFFS));

This will return data in the middle of the 'NNNNNNNNN' region above, which can be anything (but not the syscall index). This API passes these indexes to windows_version_init, which (on Windows 10) ultimately sets the global syscalls = windows_unknown_syscalls;

Further down in processing in syscalls_init, there's an explicit check against windows_unknown_syscalls, which attempts to extract all the syscall indexes on unknown windows versions. This again has a similar 'is hooked' check, and will extract incorrect values for syscall index when they are hooked like above.

What ends up failing in my case is the first call to nt_allocate_virtual_memory, because that is using the wrong syscall index. If some of the other native APIs are hooked, it might be elsewhere. In my case I saw that happen from os_heap_reserve_in_region which calls os_heap_reserve, which calls nt_allocate_virtual_memory. This returns a bad status code, which the caller thinks means the OS is out of memory. Ultimately this results in a popup saying "Out of memory. Program aborted."

Here's what the registers look like right at the syscall:

1: kd> u @rip
dynamorio!dynamorio_syscall_syscall+0x47 [D:\derek\dr\build_package\build_release-64\core\CMakeFiles\dynamorio.dir\arch\x86\x86.asm.obj.s @ 2405]:
00000000`710b7985 0f05            syscall
00000000`710b7987 c3              ret

-- syscall index is specified in eax.
1: kd> r @eax
eax=56356

1: kd> u ntdll!NtAllocateVirtualMemory
ntdll!NtAllocateVirtualMemory:
00007ffc`3ba550d0 4c8bd1          mov     r10,rcx
00007ffc`3ba550d3 e956630500      jmp     ntdll!ResCSegmentPopulate+0x39e (00007ffc`3baab42e)
00007ffc`3ba550d8 f604250803fe7f01 test    byte ptr [SharedUserData+0x308 (00000000`7ffe0308)],1
00007ffc`3ba550e0 7503            jne     ntdll!NtAllocateVirtualMemory+0x15 (00007ffc`3ba550e5)

-- syscall index taken from offset + 4
1: kd> dd ntdll!NtAllocateVirtualMemory+4 L1
00007ffc`3ba550d4  00056356

On Windows 10, due to the extra 'unknown version' handling, this exhibits as an 'out of memory' error. On Windows 7, depending on what APIs are hooked, I believe it might just crash (I don't have a Windows 7 machine convenient to test).

I believe the 'is this API hooked' logic needs to be a bit more robust, perhaps doing a full memcmp against all expected syscall stub types (masking out the bits that change, like syscall index).