Initial support of JIT/arm64

SUMMARY

We implemented a prototype of PHP JIT/arm64. Briefly speaking,

1. build system
Changes to the build system are made so that PHP JIT can be successfully
built and run on ARM-based machine.
Major change lies in file zend_jit_arm64.dasc, where the handler for
each opcode is generated into machine code. Note that this file is just
copied from zend_jit_x86.dasc and the *unimplemented* parts are
substitued with 'brk' instruction for future work.

2. registers
AArch64 registers are defined in file zend_jit_arm64.h. From our
perspectives, the register usage is quite different from the x86
implementation due to the different ABI, number of registers and
addressing modes.
We had many confusions on this part, and will discuss it in details in
the final section.

3. opcodes
Several opcodes are partially supported, including INIT_FCALL, DO_UCALL,
DO_ICALL, RETURN, ADD, PRE_INC, JMP, QM_ASSIGN, etc. Hence, simple use
scenarios such as user function call, loops, addition with integer and
floating point numbers can be supported.
18 micro test cases are added under 'ext/opcache/tests/jit/arm64/'. Note
that majority of these test cases are design for functional JIT, and
cases 'hot_func_*.phpt' and 'loop_002.phpt' can trigger tracing JIT.

4. test
Our local test environment is an ARM-based server with Ubuntu 20.04 and
GCC-10. Note that both HYBRID and CALL VM modes are supported. We
suggest running the JIT test cases using the following command. Out of
all 130 test cases, 66 cases can be passed currently.
```
  $ make test TESTS='-d opcache.jit=1203 ext/opcache/tests/jit/'
```

DETAILS

1. I-cache flush
Instruction cache must be flushed for the JIT-ed code on AArch64. See
macro JIT_CACHE_FLUSH in file 'zend_jit_internal.h'.

2. Disassembler
Add initialization and jump target parse operations for AArch64 backed.
See the updates in file 'zend_jit_disasm.c'.

3. redzone
Enable redzone for AArch64. See the update in zend_vm_opcodes.h.
Redzone is designated to prevent 'vm_stack_data' from being optimized
out by compilers. It's worth noting that this 16-byte redzone might be
reused as temporary use(treated as extra stack space) for HYBRID mode.

4. stack space reservation
The definitions of HYBRID_SPAD, SPAD and NR_SPAD are a bit tricky for
x86/64.
In AArch64, HYBRID_SPAD and SPAD are both defined as 16. These 16 bytes
are pre-allocated for tempoerary usage along the exuection of JIT-ed
code. Take line 4185 in file zend_jit_arm64.dasc as an example. NR_SPAD
is defined as 48, out of which 32 bytes to save FP/IP/LR registers.
Note that we choose to always reserve HYBRID_SPAD bytes in HYBRID mode,
no matter whether redzone is used or not, for the sake of safety.

5. stack alignment
In AArch64 the stack pointer should be 16-byte aligned. Since shadow
stack is used for JIT, it's easy to guarantee the stack alignment, via
simply moving SP with an offset like 16 or a multiple of 16. That's why
NR_SPAD is defined as 48 and we use 32 of them to save FP/IP/LR
registers which only occupies 24 bytes.

6. global registers
x27 and x28 are reserved as global registers. See the updates in file
zend_jit_vm_helpers.c

7. function prologue for CALL mode
Two callee-saved registers x27 and x28 should saved in function
zend_jit_prologue() in file zend_jit_arm64.dasc. Besides the LR, i.e.
x30, should also be saved since runtime C helper functions(such as
zend_jit_find_func_helper) might be invoked along the execution of
JIT-ed code.

8. regset
Minor changes are done to regset operations particularly for AArch64.
See the updates in file zend_jit_internal.h.

REGISTER USAGE

In this section, we will first talk about our understanding on register
usage and then demonstrate our design.

1. Register usage for HYBRID/CALL modes
Registers are used similarly between HYBRID mode and CALL mode.

One difference is how FP and IP are saved. In HYBRID mode, they are
assigned to global registers, while in CALL mode they are saved/restored
on the VM stack explicitly in prologue/epilogue.

The other difference is that LR register should also be saved/restored
in CALL mode since JIT-ed code are invoked as normal functions.

2. Register usage for functional/tracing JIT
The way registers are used differs a lot between functional JIT and
tracing JIT.

For functional JIT, runtime C code (e.g. helper functions) would be
invoked along the execution of JIT-ed code. As the operands for *most*
opcodes are accessed via the stack slot, i.e. FP + offset. Hence there
is no need to save/restore local(caller-saved) registers before/after
invoking runtime C code.
Exception lies in Phi node and registers might be allocated for these
nodes. Currently I don't fully understand the reason, why registers are
allocated for Phi functions, because I suppose for different versions of
SSA variables at the Phi function, their postions on the stack slot
should be identical(in other words, access via the stack slot is enough
and there is no need to allocate registers).

For tracing JIT, runtime information are recorded for traces(before the
JIT compilation), and the data types and control flows are concrete as
well. Hence it's would be faster to conduct operations and computations
via registers rather than stack slots(as functional JIT does) for these
collected hot paths. Besides, runtime C code can be invoked for tracing
JIT, however this only happends for deoptimization and all registers are
saved to stack in advance.

3. Candidates for register allocator
1) opcode candidates
Function zend_jit_opline_supports_reg() determines the candidate opcodes
which can use CPU registers.

2) register candidates
Registers in set "ZEND_REGSET_FP + ZEND_REGSET_GP - ZEND_REGSET_FIXED -
ZEND_REGSET_PRESERVED" are available for register allocator.
Note that registers from ZEND_REGSET_FIXED are reserved for special
purpose, such as the stack pointer, and they are excluded from register
allocation process.
Note that registers from ZEND_REGSET_PRESERVED are callee-saved based on
the ABI and it's safe to not use them either.

4. Temporary registers
Temporary registers are needed by some opcodes to save intermediate
computation results.

1) Functions zend_jit_get_def_scratch_regset() and
zend_jit_get_scratch_regset() return which registers might be clobbered
by some opcodes. Hence register allocator would spill these scratch
registers if necessary when encountering these opcodes.

2) Macro ZEND_REGSET_LOW_PRIORITY denotes a set of registers which would
be allocated with low priority, and these registers can be used as
temporary usage to avoid conflicts to its best.

5. Compared to the x86 implementation, in JIT/arm64
1) Called-saved FP registers are included into ZEND_REGSET_PRESERVED for
AArch64.

2) We follow the logic of function zend_jit_opline_supports_reg().

3) We reserve 4 GPRs and 2 FPRs out from register allocator and use them
as temporary registers in particular. Note that these 6 registers are
included in set ZEND_REGSET_FIXED.
Since they are reserved, may-clobbered registers can be removed for most
opcodes except for function calls. Besides, low-priority registers are
defined as empty since all candidate registers are of the same priority.
See the updates in function zend_jit_get_scratch_regset() and macro
ZEND_REGSET_LOW_PRIORITY.

6. Why we reserve registers for temporary usage?
1) Addressing mode in AArch64 needs more temporary registers.
The addressing mode is different from x86 and tempory registers might be
*always* needed for most opcodes. For instance, an immediate must be
first moved into one register before storing into memory in AArch64,
whereas in x86 this immediate can be stored directly.

2) There are more registers in AArch64.
Compared to the solution in JIT/x86(that is, temporary registers are
reserved on demand, i.e. different registers for different opcodes under
different conditions), our solution seems a coarse-granularity and
brute-force solution, and the execution performance might be downgraded
to some extent since the number of candidate registers used for
allocation becomes less.
We suppose the performance loss might be acceptable since there are more
registers in AArch64.

3) Based on my understanding, scratch registers defined in x86 are
excluded from candidates for register allocator with *low possibility*,
and it can still allocate these registers. Special handling should be
conducted, such as checking 'reg != ZREG_R0'.
Hence, as we see it, it's simpler to reserve some temporary registers
exclusively. See the updates in function zend_jit_math_long_long() for
instance. TMP1 can be used directly without checking.

Co-Developed-by: Nick Gasson <Nick.Gasson@arm.com>

Zend/zend_vm_opcodes.h

@@ -35,7 +35,8 @@
 #endif
 
 #if (ZEND_VM_KIND == ZEND_VM_KIND_HYBRID) && !defined(__SANITIZE_ADDRESS__)
-# if ((defined(i386) && !defined(__PIC__)) || defined(__x86_64__) || defined(_M_X64))
+# if ((defined(i386) && !defined(__PIC__)) || defined(__x86_64__) || \
+       defined(_M_X64) || defined(__aarch64__))
 #  define ZEND_VM_HYBRID_JIT_RED_ZONE_SIZE 16
 # endif
 #endif

build/Makefile.global

@@ -125,6 +125,8 @@ distclean: clean
 	rm -f scripts/man1/phpize.1 scripts/php-config scripts/man1/php-config.1 sapi/cli/php.1 sapi/cgi/php-cgi.1 sapi/phpdbg/phpdbg.1 ext/phar/phar.1 ext/phar/phar.phar.1
 	rm -f sapi/fpm/php-fpm.conf sapi/fpm/init.d.php-fpm sapi/fpm/php-fpm.service sapi/fpm/php-fpm.8 sapi/fpm/status.html
 	rm -f ext/phar/phar.phar ext/phar/phar.php
+	rm -f ext/opcache/jit/zend_jit_x86.c
+	rm -f ext/opcache/jit/zend_jit_arm64.c
 	if test "$(srcdir)" != "$(builddir)"; then \
 	  rm -f ext/phar/phar/phar.inc; \
 	fi

ext/opcache/config.m4

@@ -29,7 +29,7 @@ if test "$PHP_OPCACHE" != "no"; then
 
   if test "$PHP_OPCACHE_JIT" = "yes"; then
     case $host_cpu in
-      i[[34567]]86*|x86*)
+      i[[34567]]86*|x86*|aarch64)
         ;;
       *)
         AC_MSG_WARN([JIT not supported by host architecture])
@@ -77,6 +77,7 @@ if test "$PHP_OPCACHE" != "no"; then
     fi
 
     PHP_SUBST(DASM_FLAGS)
+    PHP_SUBST(DASM_ARCH)
 
     AC_MSG_CHECKING(for opagent in default path)
     for i in /usr/local /usr; do

ext/opcache/config.w32

@@ -25,6 +25,7 @@ if (PHP_OPCACHE != "no") {
 				dasm_flags += " -D ZTS=1";
 			}
 			DEFINE("DASM_FLAGS", dasm_flags);
+			DEFINE("DASM_ARCH", "x86");
 
 			AC_DEFINE('HAVE_JIT', 1, 'Define to enable JIT');
 			/* XXX read this dynamically */

ext/opcache/jit/Makefile.frag

@@ -2,11 +2,11 @@
 $(builddir)/minilua: $(srcdir)/jit/dynasm/minilua.c
 	$(BUILD_CC) $(srcdir)/jit/dynasm/minilua.c -lm -o $@
 
-$(builddir)/jit/zend_jit_x86.c: $(srcdir)/jit/zend_jit_x86.dasc $(srcdir)/jit/dynasm/*.lua $(builddir)/minilua
-	$(builddir)/minilua $(srcdir)/jit/dynasm/dynasm.lua  $(DASM_FLAGS) -o $@ $(srcdir)/jit/zend_jit_x86.dasc
+$(builddir)/jit/zend_jit_$(DASM_ARCH).c: $(srcdir)/jit/zend_jit_$(DASM_ARCH).dasc $(srcdir)/jit/dynasm/*.lua $(builddir)/minilua
+	$(builddir)/minilua $(srcdir)/jit/dynasm/dynasm.lua  $(DASM_FLAGS) -o $@ $(srcdir)/jit/zend_jit_$(DASM_ARCH).dasc
 
 $(builddir)/jit/zend_jit.lo: \
-	$(builddir)/jit/zend_jit_x86.c \
+	$(builddir)/jit/zend_jit_$(DASM_ARCH).c \
 	$(srcdir)/jit/zend_jit_helpers.c \
 	$(srcdir)/jit/zend_jit_disasm.c \
 	$(srcdir)/jit/zend_jit_gdb.c \

ext/opcache/jit/zend_jit.c

@@ -39,7 +39,14 @@
 #include "Optimizer/zend_call_graph.h"
 #include "Optimizer/zend_dump.h"
 
+#if defined(__x86_64__) || defined(i386)
 #include "jit/zend_jit_x86.h"
+#elif defined (__aarch64__)
+#include "jit/zend_jit_arm64.h"
+#else
+#error "JIT not supported on this platform"
+#endif
+
 #include "jit/zend_jit_internal.h"
 
 #ifdef ZTS
@@ -204,7 +211,12 @@ static bool zend_long_is_power_of_two(zend_long x)
 #define OP2_RANGE()      OP_RANGE(ssa_op, op2)
 #define OP1_DATA_RANGE() OP_RANGE(ssa_op + 1, op1)
 
+#if defined(__x86_64__) || defined(i386)
 #include "dynasm/dasm_x86.h"
+#elif defined(__aarch64__)
+#include "dynasm/dasm_arm64.h"
+#endif
+
 #include "jit/zend_jit_helpers.c"
 #include "jit/zend_jit_disasm.c"
 #ifndef _WIN32
@@ -216,7 +228,11 @@ static bool zend_long_is_power_of_two(zend_long x)
 #endif
 #include "jit/zend_jit_vtune.c"
 
+#if defined(__x86_64__) || defined(i386)
 #include "jit/zend_jit_x86.c"
+#elif defined(__aarch64__)
+#include "jit/zend_jit_arm64.c"
+#endif
 
 #if _WIN32
 # include <Windows.h>
@@ -298,15 +314,32 @@ static void handle_dasm_error(int ret) {
 		case DASM_S_RANGE_PC:
 			fprintf(stderr, "DASM_S_RANGE_PC %d\n", ret & 0xffffffu);
 			break;
+#ifdef DASM_S_RANGE_VREG
 		case DASM_S_RANGE_VREG:
 			fprintf(stderr, "DASM_S_RANGE_VREG\n");
 			break;
+#endif
+#ifdef DASM_S_UNDEF_L
 		case DASM_S_UNDEF_L:
 			fprintf(stderr, "DASM_S_UNDEF_L\n");
 			break;
+#endif
+#ifdef DASM_S_UNDEF_LG
+		case DASM_S_UNDEF_LG:
+			fprintf(stderr, "DASM_S_UNDEF_LG\n");
+			break;
+#endif
+#ifdef DASM_S_RANGE_REL
+		case DASM_S_RANGE_REL:
+			fprintf(stderr, "DASM_S_RANGE_REL\n");
+			break;
+#endif
 		case DASM_S_UNDEF_PC:
 			fprintf(stderr, "DASM_S_UNDEF_PC\n");
 			break;
+		default:
+			fprintf(stderr, "DASM_S_%0x\n", ret & 0xff000000u);
+			break;
 	}
 	ZEND_UNREACHABLE();
 }
@@ -391,6 +424,9 @@ static void *dasm_link_and_encode(dasm_State             **dasm_state,
 	entry = *dasm_ptr;
 	*dasm_ptr = (void*)((char*)*dasm_ptr + ZEND_MM_ALIGNED_SIZE_EX(size, DASM_ALIGNMENT));
 
+	/* flush the hardware I-cache */
+	JIT_CACHE_FLUSH(entry, entry + size);
+
 	if (trace_num) {
 		zend_jit_trace_add_code(entry, size);
 	}

ext/opcache/jit/zend_jit_arm64.h

@@ -0,0 +1,142 @@
+/*
+   +----------------------------------------------------------------------+
+   | Zend JIT                                                             |
+   +----------------------------------------------------------------------+
+   | Copyright (c) The PHP Group                                          |
+   +----------------------------------------------------------------------+
+   | This source file is subject to version 3.01 of the PHP license,      |
+   | that is bundled with this package in the file LICENSE, and is        |
+   | available through the world-wide-web at the following url:           |
+   | http://www.php.net/license/3_01.txt                                  |
+   | If you did not receive a copy of the PHP license and are unable to   |
+   | obtain it through the world-wide-web, please send a note to          |
+   | license@php.net so we can mail you a copy immediately.               |
+   +----------------------------------------------------------------------+
+   | Authors: Dmitry Stogov <dmitry@php.net>                              |
+   |          Hao Sun <hao.sun@arm.com>                                   |
+   +----------------------------------------------------------------------+
+*/
+
+#ifndef HAVE_JIT_ARM64_H
+#define HAVE_JIT_ARM64_H
+
+typedef enum _zend_reg {
+	ZREG_NONE = -1,
+
+	ZREG_X0,
+	ZREG_X1,
+	ZREG_X2,
+	ZREG_X3,
+	ZREG_X4,
+	ZREG_X5,
+	ZREG_X6,
+	ZREG_X7,
+	ZREG_X8,
+	ZREG_X9,
+	ZREG_X10,
+	ZREG_X11,
+	ZREG_X12,
+	ZREG_X13,
+	ZREG_X14,
+	ZREG_X15,
+	ZREG_X16,
+	ZREG_X17,
+	ZREG_X18,
+	ZREG_X19,
+	ZREG_X20,
+	ZREG_X21,
+	ZREG_X22,
+	ZREG_X23,
+	ZREG_X24,
+	ZREG_X25,
+	ZREG_X26,
+	ZREG_X27,
+	ZREG_X28,
+	ZREG_X29,
+	ZREG_X30,
+	ZREG_X31,
+
+	ZREG_V0,
+	ZREG_V1,
+	ZREG_V2,
+	ZREG_V3,
+	ZREG_V4,
+	ZREG_V5,
+	ZREG_V6,
+	ZREG_V7,
+	ZREG_V8,
+	ZREG_V9,
+	ZREG_V10,
+	ZREG_V11,
+	ZREG_V12,
+	ZREG_V13,
+	ZREG_V14,
+	ZREG_V15,
+	ZREG_V16,
+	ZREG_V17,
+	ZREG_V18,
+	ZREG_V19,
+	ZREG_V20,
+	ZREG_V21,
+	ZREG_V22,
+	ZREG_V23,
+	ZREG_V24,
+	ZREG_V25,
+	ZREG_V26,
+	ZREG_V27,
+	ZREG_V28,
+	ZREG_V29,
+	ZREG_V30,
+	ZREG_V31,
+
+	ZREG_NUM,
+
+	ZREG_THIS, /* used for delayed FETCH_THIS deoptimization */
+
+	/* pseudo constants used by deoptimizer */
+	ZREG_LONG_MIN_MINUS_1,
+	ZREG_LONG_MIN,
+	ZREG_LONG_MAX,
+	ZREG_LONG_MAX_PLUS_1,
+	ZREG_NULL,
+
+	ZREG_ZVAL_TRY_ADDREF,
+	ZREG_ZVAL_COPY_GPR0,
+} zend_reg;
+
+typedef struct _zend_jit_registers_buf {
+	uint64_t gpr[32]; /* general purpose integer register */
+	double fpr[32];   /* floating point registers */
+} zend_jit_registers_buf;
+
+#define ZREG_FIRST_FPR ZREG_V0
+
+#define ZREG_RSP ZREG_X31
+#define ZREG_RLR ZREG_X30
+#define ZREG_RFP ZREG_X29
+#define ZREG_RPR ZREG_X18
+
+# define ZREG_FP      ZREG_X27
+# define ZREG_IP      ZREG_X28
+# define ZREG_RX      ZREG_IP
+
+typedef uint64_t zend_regset;
+
+# define ZEND_REGSET_FIXED \
+	(ZEND_REGSET(ZREG_RSP) | ZEND_REGSET(ZREG_RLR) | ZEND_REGSET(ZREG_RFP) | \
+	ZEND_REGSET(ZREG_RPR) | ZEND_REGSET(ZREG_FP) | ZEND_REGSET(ZREG_IP) | \
+	ZEND_REGSET_INTERVAL(ZREG_X8, ZREG_X11) | \
+	ZEND_REGSET_INTERVAL(ZREG_V16, ZREG_V17))
+# define ZEND_REGSET_GP \
+	ZEND_REGSET_DIFFERENCE(ZEND_REGSET_INTERVAL(ZREG_X0, ZREG_X30), ZEND_REGSET_FIXED)
+# define ZEND_REGSET_FP \
+	ZEND_REGSET_DIFFERENCE(ZEND_REGSET_INTERVAL(ZREG_V0, ZREG_V31), ZEND_REGSET_FIXED)
+# define ZEND_REGSET_SCRATCH \
+	(ZEND_REGSET_INTERVAL(ZREG_X0, ZREG_X17) | ZEND_REGSET_INTERVAL(ZREG_V0, ZREG_V7) | \
+	ZEND_REGSET_INTERVAL(ZREG_V16, ZREG_V31))
+# define ZEND_REGSET_PRESERVED \
+	(ZEND_REGSET_INTERVAL(ZREG_X19, ZREG_X28) | ZEND_REGSET_INTERVAL(ZREG_V8, ZREG_V15))
+
+#define ZEND_REGSET_LOW_PRIORITY ZEND_REGSET_EMPTY
+
+#endif /* ZEND_JIT_ARM64_H */

ext/opcache/jit/zend_jit_disasm.c

@@ -225,6 +225,7 @@ static uint64_t zend_jit_disasm_branch_target(csh cs, const cs_insn *insn)
 {
 	unsigned int i;
 
+#if defined(__x86_64__) || defined(i386)
 	if (cs_insn_group(cs, insn, X86_GRP_JUMP)) {
 		for (i = 0; i < insn->detail->x86.op_count; i++) {
 			if (insn->detail->x86.operands[i].type == X86_OP_IMM) {
@@ -232,6 +233,14 @@ static uint64_t zend_jit_disasm_branch_target(csh cs, const cs_insn *insn)
 			}
 		}
 	}
+#elif defined(__aarch64__)
+	if (cs_insn_group(cs, insn, ARM64_GRP_JUMP)) {
+		for (i = 0; i < insn->detail->arm64.op_count; i++) {
+			if (insn->detail->arm64.operands[i].type == ARM64_OP_IMM)
+				return insn->detail->arm64.operands[i].imm;
+		}
+	}
+#endif
 
 	return 0;
 }
@@ -319,6 +328,11 @@ static int zend_jit_disasm(const char    *name,
 #  else
 	cs_option(cs, CS_OPT_SYNTAX, CS_OPT_SYNTAX_ATT);
 #  endif
+# elif defined(__aarch64__)
+	if (cs_open(CS_ARCH_ARM64, CS_MODE_ARM, &cs) != CS_ERR_OK)
+		return 0;
+	cs_option(cs, CS_OPT_DETAIL, CS_OPT_ON);
+	cs_option(cs, CS_OPT_SYNTAX, CS_OPT_SYNTAX_ATT);
 # else
 	if (cs_open(CS_ARCH_X86, CS_MODE_32, &cs) != CS_ERR_OK)
 		return 0;
@@ -431,7 +445,11 @@ static int zend_jit_disasm(const char    *name,
 		}
 
 # ifdef HAVE_CAPSTONE_ITER
+#  if defined(__aarch64__)
+		fprintf(stderr, "    "ZEND_XLONG_FMT":\t%s ", insn->address, insn->mnemonic);
+#  else
 		fprintf(stderr, "\t%s ", insn->mnemonic);
+#  endif
 		p = insn->op_str;
 # else
 		fprintf(stderr, "\t%s ", insn[i].mnemonic);

ext/opcache/jit/zend_jit_gdb.c

@@ -20,7 +20,12 @@
    +----------------------------------------------------------------------+
 */
 
+
+#if defined(__x86_64__) || defined(i386)
 #define HAVE_GDB
+#else
+#warning Missing GDB JIT support on this platform
+#endif
 
 #ifdef HAVE_GDB
 

ext/opcache/jit/zend_jit_internal.h

@@ -14,6 +14,7 @@
    +----------------------------------------------------------------------+
    | Authors: Dmitry Stogov <dmitry@php.net>                              |
    |          Xinchen Hui <laruence@php.net>                              |
+   |          Hao Sun <hao.sun@arm.com>                                   |
    +----------------------------------------------------------------------+
 */
 
@@ -26,11 +27,24 @@
 #define ZEND_REGSET_IS_EMPTY(regset) \
 	(regset == ZEND_REGSET_EMPTY)
 
+#define ZEND_REGSET_IS_SINGLETON(regset) \
+	(regset && !(regset & (regset - 1)))
+
+#if (ZREG_NUM <= 32)
 #define ZEND_REGSET(reg) \
 	(1u << (reg))
+#else
+#define ZEND_REGSET(reg) \
+	(1ull << (reg))
+#endif
 
+#if (ZREG_NUM <= 32)
 #define ZEND_REGSET_INTERVAL(reg1, reg2) \
 	(((1u << ((reg2) - (reg1) + 1)) - 1) << (reg1))
+#else
+#define ZEND_REGSET_INTERVAL(reg1, reg2) \
+	(((1ull << ((reg2) - (reg1) + 1)) - 1) << (reg1))
+#endif
 
 #define ZEND_REGSET_IN(regset, reg) \
 	(((regset) & ZEND_REGSET(reg)) != 0)
@@ -50,7 +64,11 @@
 #define ZEND_REGSET_DIFFERENCE(set1, set2) \
 	((set1) & ~(set2))
 
-#ifndef _WIN32
+#if defined (__aarch64__)
+# define ZEND_REGSET_FIRST(set)  ((zend_reg)__builtin_ctzll(set))
+# define ZEND_REGSET_SECOND(set) ((zend_reg)__builtin_ctzll(set ^ (1ull << ZEND_REGSET_FIRST(set))))
+# define ZEND_REGSET_LAST(set)   ((zend_reg)(__builtin_clzll(set)^63))
+#elif !defined(_WIN32)
 # if (ZREG_NUM <= 32)
 #  define ZEND_REGSET_FIRST(set) ((zend_reg)__builtin_ctz(set))
 #  define ZEND_REGSET_LAST(set)  ((zend_reg)(__builtin_clz(set)^31))
@@ -711,4 +729,17 @@ static zend_always_inline bool zend_jit_may_be_polymorphic_call(const zend_op *o
 	}
 }
 
+/* Instruction cache flush */
+#ifndef JIT_CACHE_FLUSH
+#  if defined (__aarch64__)
+#    if ((defined(__GNUC__) && ZEND_GCC_VERSION >= 4003) || __has_builtin(__builtin___clear_cache))
+#      define JIT_CACHE_FLUSH(from, to) __builtin___clear_cache((char*)(from), (char*)(to))
+#    else
+#      error "Missing builtin to flush instruction cache for AArch64"
+#    endif
+#  else /* Not required to implement on archs with unified caches */
+#    define JIT_CACHE_FLUSH(from, to)
+#  endif
+#endif /* !JIT_CACHE_FLUSH */
+
 #endif /* ZEND_JIT_INTERNAL_H */

ext/opcache/jit/zend_jit_perf_dump.c

@@ -22,6 +22,9 @@
 #include <unistd.h>
 #include <time.h>
 #include <sys/mman.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
 
 #if defined(__linux__)
 #include <sys/syscall.h>

ext/opcache/jit/zend_jit_vm_helpers.c

@@ -28,7 +28,12 @@
 #include "Optimizer/zend_func_info.h"
 #include "Optimizer/zend_call_graph.h"
 #include "zend_jit.h"
+#if defined(__x86_64__) || defined(i386)
 #include "zend_jit_x86.h"
+#elif defined(__aarch64__)
+#include "zend_jit_arm64.h"
+#endif
+
 #include "zend_jit_internal.h"
 
 #ifdef HAVE_GCC_GLOBAL_REGS
@@ -36,9 +41,12 @@
 # if defined(__x86_64__)
 register zend_execute_data* volatile execute_data __asm__("%r14");
 register const zend_op* volatile opline __asm__("%r15");
-# else
+# elif defined(i386)
 register zend_execute_data* volatile execute_data __asm__("%esi");
 register const zend_op* volatile opline __asm__("%edi");
+# elif defined(__aarch64__)
+register zend_execute_data* volatile execute_data __asm__("x27");
+register const zend_op* volatile opline __asm__("x28");
 # endif
 # pragma GCC diagnostic warning "-Wvolatile-register-var"
 #endif

ext/opcache/tests/jit/arm64/add_001.phpt

@@ -0,0 +1,20 @@
+--TEST--
+JIT ADD: 001
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo($var) {
+  $res = $var + 1;
+  var_dump($res);
+}
+foo(1);
+?>
+--EXPECT--
+int(2)

ext/opcache/tests/jit/arm64/add_002.phpt

@@ -0,0 +1,20 @@
+--TEST--
+JIT ADD: 002
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo($var) {
+  $res = $var + 0x1000;
+  var_dump($res);
+}
+foo(1);
+?>
+--EXPECT--
+int(4097)

ext/opcache/tests/jit/arm64/add_003.phpt

@@ -0,0 +1,20 @@
+--TEST--
+JIT ADD: 003
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo($var) {
+  $ret = $var + 1;
+  var_dump($ret);
+}
+foo(PHP_INT_MAX);
+?>
+--EXPECT--
+float(9.223372036854776E+18)

ext/opcache/tests/jit/arm64/add_004.phpt

@@ -0,0 +1,20 @@
+--TEST--
+JIT ADD: 004
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo($var) {
+  $ret = $var + 200;
+  var_dump($ret);
+}
+foo(PHP_INT_MAX);
+?>
+--EXPECT--
+float(9.223372036854776E+18)

ext/opcache/tests/jit/arm64/add_005.phpt

@@ -0,0 +1,24 @@
+--TEST--
+JIT ADD: 005
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo($var) {
+  $res = $var + 2;
+  var_dump($res);
+}
+foo("hello");
+?>
+--EXPECTF--
+Fatal error: Uncaught TypeError: Unsupported operand types: string + int in %s:%d
+Stack trace:
+#0 %s(%d): foo('hello')
+#1 {main}
+  thrown in %s on line %d

ext/opcache/tests/jit/arm64/hot_func_001.phpt

@@ -0,0 +1,24 @@
+--TEST--
+JIT HOT_FUNC: 001
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+opcache.jit=tracing
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo() {
+  var_dump("hello");
+}
+foo();
+foo();
+foo();
+?>
+--EXPECT--
+string(5) "hello"
+string(5) "hello"
+string(5) "hello"

ext/opcache/tests/jit/arm64/hot_func_002.phpt

@@ -0,0 +1,25 @@
+--TEST--
+JIT HOT_FUNC: 002
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+opcache.jit=tracing
+opcache.jit_hot_func=2
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo() {
+  var_dump("hello");
+}
+foo();
+foo();
+foo();
+?>
+--EXPECT--
+string(5) "hello"
+string(5) "hello"
+string(5) "hello"

ext/opcache/tests/jit/arm64/icall_001.phpt

@@ -0,0 +1,24 @@
+--TEST--
+JIT ICALL: 001
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=1M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+var_dump(true, 0, 42, -42, 0.0, 2.0,"hello", array());
+?>
+--EXPECT--
+bool(true)
+int(0)
+int(42)
+int(-42)
+float(0)
+float(2)
+string(5) "hello"
+array(0) {
+}

ext/opcache/tests/jit/arm64/loop_001.phpt

@@ -0,0 +1,22 @@
+--TEST--
+JIT LOOP: 001
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo($n) {
+  $res = 0;
+  for ($i = 1; $i < $n; $i++)
+    $res = $res + $i;
+  return $res;
+}
+print foo(5);
+?>
+--EXPECT--
+10

ext/opcache/tests/jit/arm64/loop_002.phpt

@@ -0,0 +1,26 @@
+--TEST--
+JIT HOT LOOP: 002
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+opcache.jit=tracing
+opcache.jit_hot_func=2
+opcache.jit_hot_loop=2
+opcache.jit_hot_side_exit=0
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo($n) {
+  $res = 0;
+  for ($i = 1; $i < $n; $i++)
+    $res = $res + $i;
+  return $res;
+}
+print foo(5);
+?>
+--EXPECT--
+10

ext/opcache/tests/jit/arm64/recv_001.phpt

@@ -0,0 +1,26 @@
+--TEST--
+JIT RECV: 001
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo($var) {
+  var_dump($var);
+}
+foo(1);
+foo(1.0);
+foo("hello");
+foo(array());
+?>
+--EXPECT--
+int(1)
+float(1)
+string(5) "hello"
+array(0) {
+}

ext/opcache/tests/jit/arm64/ret_001.phpt

@@ -0,0 +1,20 @@
+--TEST--
+JIT RET: 001
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo() {
+  $c = 1;
+  return $c;
+}
+var_dump(foo());
+?>
+--EXPECT--
+int(1)

ext/opcache/tests/jit/arm64/ret_002.phpt

@@ -0,0 +1,20 @@
+--TEST--
+JIT RET: 002
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo() {
+  $c = 1.0;
+  return $c;
+}
+var_dump(foo());
+?>
+--EXPECT--
+float(1)

ext/opcache/tests/jit/arm64/ret_003.phpt

@@ -0,0 +1,20 @@
+--TEST--
+JIT RET: 003
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo() {
+  $c = "hello";
+  return $c;
+}
+var_dump(foo());
+?>
+--EXPECT--
+string(5) "hello"

ext/opcache/tests/jit/arm64/skipif.inc

@@ -0,0 +1,3 @@
+<?php
+  if (!extension_loaded('Zend OPcache')) die('skip Zend OPcache extension not available');
+?>

ext/opcache/tests/jit/arm64/ucall_001.phpt

@@ -0,0 +1,19 @@
+--TEST--
+JIT UCALL: 001
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo() {
+    var_dump("hello");
+}
+foo();
+?>
+--EXPECT--
+string(5) "hello"

ext/opcache/tests/jit/arm64/ucall_002.phpt

@@ -0,0 +1,21 @@
+--TEST--
+JIT UCALL: 002
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo() {
+    var_dump("hello");
+}
+foo();
+var_dump("world!");
+?>
+--EXPECT--
+string(5) "hello"
+string(6) "world!"

ext/opcache/tests/jit/arm64/ucall_003.phpt

@@ -0,0 +1,22 @@
+--TEST--
+JIT UCALL: 003
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo() {
+  var_dump("hello");
+}
+function bar() {
+  foo();
+}
+bar();
+?>
+--EXPECT--
+string(5) "hello"

ext/opcache/tests/jit/arm64/ucall_004.phpt

@@ -0,0 +1,23 @@
+--TEST--
+JIT UCALL: 004
+--INI--
+opcache.enable=1
+opcache.enable_cli=1
+opcache.file_update_protection=0
+opcache.jit_buffer_size=32M
+;opcache.jit_debug=257
+--SKIPIF--
+<?php require_once('skipif.inc'); ?>
+--FILE--
+<?php
+function foo() {
+  var_dump("hello");
+}
+foo();
+foo();
+foo();
+?>
+--EXPECT--
+string(5) "hello"
+string(5) "hello"
+string(5) "hello"