This test case did 100,000 includes of a non existing file to show the
memory leak; this is not necessary, because after the first failing
include, memory is not supposed to increase for further includes.
Closes GH-7088.
Currently interface methods with visibility `private` or `protected` fail
with an error message:
Access type for interface method A::b() must be omitted
However, explicitly setting visibility `public` is allowed and often desired.
This commit updates the error message to:
Access type for interface method A::b() must be public
* JIT/AArch64: Code refactoring for macros
Update the comments and rename arguments for some macros.
The following macros are renamed:
SAFE_MEM_ACC_WITH_UOFFSET -> MEM_ACC_64_WITH_UOFFSET
SAFE_MEM_ACC_WITH_UOFFSET_32 -> MEM_ACC_32_WITH_UOFFSET
SAFE_MEM_ACC_WITH_UOFFSET_BYTE -> MEM_ACC_8_WITH_UOFFSET
MEM_STORE_BYTE_ZTS -> MEM_STORE_8_ZTS
MEM_STORE_ZTS -> MEM_STORE_64_ZTS
MEM_LOAD_ZTS -> MEM_LOAD_64_ZTS
MEM_LOAD_BYTE_ZTS -> MEM_LOAD_8_ZTS
ADD_IP_FROM_CST -> ADD_IP_WITH_CONST
The following macros are deprecated and removed:
SET_Z_PTR, GET_Z_W2, SET_Z_W2
Change-Id: I767cf70f373e5f5a1090079e70f8e953a654da00
* Use MEM_ACCESS_*_WITH_UOFFSET
It's more accurate to use "MEM_ACCESS_*_WITH_UOFFSET" than
"MEM_ACC_*_WITH_UOFFSET".
Change-Id: I71479a809008848b61c4786016e6c10110e6aa8b
* Revert the updates for macros ADD_IP_WITH_CONST and LONG_ADD_SUB_WITH_IMM
As pointed by Dmitry, (int32_t) (1LL << 32) would make the assertion
"TRUE", which is not expected by us.
Change-Id: I767cf70f373e5f5a1090079e70f8e953a654da00
Move this code directly into the error handler, and check the
heap->overflow flag. Discarding output here allows us to print
the normal memory limit message to standard output. Otherwise
nothing would be printed unless a different log medium was used,
which makes for a suboptimal debugging experience.
As PHP has a minimum memory usage of 2M (size of allocator chunk),
setting a limit below that value is not meaningful and will be
automatically rounded up to the chunk size. Rather than doing this
silently, show the newly introduced error message.
The memory limit had to be increased to 2M for a number of tests.
tests/lang/bug45392 has been marked as XFAIL. This old bugfix is
not working as intended. The memory limit in main's `PG(memory_limit)`
differs from the one in zend_alloc. In zend_alloc the `AG(mm_heap)->limit`
is defined as `max(passed_value, ZEND_MM_CHUNK_SIZE)`. The check made in
an unclean shutdown will never be true unless the memory limit is lower
than ZEND_MM_CHUNK_SIZE, which happened to be the case in the test.
https://bugs.php.net/bug.php?id=45392https://github.com/php/php-src/commit/fcc0fdd125fdb9e1713f91d027fe07d680a0cf36
When the memory limit is reduced using an `ini_set("memory_limit", ..)`
below the currently allocated memory, the out-of-memory check overflowed.
Instead of implementing additional checks during allocation,
`zend_set_memory_limit()` now validates the new memory limit. When
below the current memory usage the ini_set call will fail and throw
a warning.
This is part of GH-7040.
If the object is not created through PDO::prepare(), e.g. in a
mock scenario, it should still be possible to initialize the
$queryString property.
See bug #81084.
This is not guaranteed to work, since the actual server name may only
be given as SAN. Since we're doing the peer verification later anyway
(using the respective context options as appropriate), there is no need
to even supply a server name when verifying against the Windows cert
store.
Closes GH-7060.
For concatenation, the in-place variant can be much more efficient,
because it will reallocate the string in-place. Special-case the
typed property compound assignment code for the case where we
concatenate to a string, in which case we know that the result
will also be a string, and we don't need the type check anyway.
Zero register, i.e. xzr, can be used directly to zero FP register.
TODO: FMOV from ZR may be slow on some cores and the preferred
instructio is MOVI with immediate zero [1]. However, MOVI is not
recoginized by DynASM/arm64.
[1] https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=523d72071960
Change-Id: I0eaee4445e05adb45c6bb80ddb62ea02cdc9f4db
Newer MSVC versions support the `inline` keyword, so we do not need to explicitly define it. We also shouldn't define `inline` to `__forceinline`, as we already have `zend_always_inline` to indicate this requirement.
* JIT/AArch64: [macos][ZTS] Support fast path for tlv_get_addr
Access to TLV(thread local variable) in macOS is in "dynamic" form and
function tlv_get_addr() is invoked to resolve the address. See the
example in [1].
Note there is one struct TLVDescriptor [2] for each TLV. The first
member holds the address of function tlv_get_addr(), and the other two
members, "key" and "offset", would be used inside tlv_get_addr().
The disassembly code for function tlv_get_addr() is shown in [3]. With
the value from system register, i.e. tpidrro_el0, together with "key"
and "offset", the TLV address can be obtained.
Note that the value from tpidrro_el0 varies for different threads, and
unique address for TLV is resolved.
It's worth noting that slow path would be executed, i.e. function
tlv_allocate_and_initialize_for_key(), for the first time of TLV access.
In this patch:
1. "_tsrm_ls_cache" is guaranteed to be accessed before any VM/JIT code
during the request startup, e.g. in init_executor(), therefore, slow
path can be avoided.
2. As TLVDecriptor is immutable and zend_jit_setup() executes once, we
get this structure in tsrm_get_ls_cache_tcb_offset(). Note the 'ldr'
instruction would be patched to 'add' by the linker.
3. Only fast path for tlv_get_addr() is implemented in macro
LOAD_TSRM_CACHE.
With this patch, all ~4k test cases can pass for ZTS+CALL in macOS on
Apple silicon.
[1] https://gist.github.com/shqking/4aab67e0105f7c1f2c549d57d5799f94
[2]
https://opensource.apple.com/source/dyld/dyld-195.6/src/threadLocalVariables.c.auto.html
[3] https://gist.github.com/shqking/329d7712c26bad49786ab0a544a4af43
Change-Id: I613e9c37e3ff2ecc3fab0f53f1e48a0246e12ee3
Excerpt from the release news:
Version 10.37 26-May-2021
-------------------------
A few more bug fixes and tidies. The only change of real note is the removal of
the actual POSIX names regcomp etc. from the POSIX wrapper library because
these have caused issues for some applications (see 10.33 #2 below).
Version 10.36 04-December-2020
------------------------------
Again, mainly bug fixes and tidies. The only enhancements are the addition of
GNU grep's -m (aka --max-count) option to pcre2grep, and also unifying the
handling of substitution strings for both -O and callouts in pcre2grep, with
the addition of $x{...} and $o{...} to allow for characters whose code points
are greater than 255 in Unicode mode.
NOTE: there is an outstanding issue with JIT support for MacOS on arm64
hardware. For details, please see Bugzilla issue #2618.
Signed-off-by: Anatol Belski <ab@php.net>
This format matches against null bytes, and prevents the test
expectation from being interpreted as binary data.
bless_tests.php will automatically replace \0 with %0 as well.
While reading the docs in the repo, I found & made a few improvements
to the documentation so it's clearer to its readers.
These improvements are around: typos, general punctuations, and grammar.
This tests takes about 2 minutes on AppVeyor CI, what is super slow.
The problem is that we're doing 50,000 inserts of small keys and values
instead of only few inserts with large values, what basically has the
same effect regarding the mmap size.
Closes GH-7073.
Now that inheritance can throw deprecations again, these may be
converted to exception by a custom error handler. In this case
we need to convert the exception to a fatal error, as inheritance
cannot safely throw in the general case.
- make_fcontext() never returns NULL.
- Show syscall error info in exception just like zend_alloc does (we also used php_win32_error_*() in Zend).
- MAP_ANON was marked as deprecated or compatibility macro on Linux, It would be better to use MAP_ANONYMOUS first.
- Makes the code consistent with the code of other modules in the kernel.
- adds some comments.
When row data split across multiple packets, allocate a temporary
buffer that can be reallocated, and only copy into the row buffer
pool arena once we know the final size. This avoids quadratic
memory usage for very large results.
(cherry picked from commit 1fc4c89214)
The order of the tokens in zend_language_parser.h depends on the
used bison version. To avoid spurious changes to tokenizer_data.c
generate tit from the zend_language_parser.y file instead.
Several extension test suites have already been changed to rely on the
new behavior of `--EXTENSIONS--` section, i.e. to dynamically load
required extensions on demand. Thus, there is no need to load these
extensions always, which should improve the performance of executing
the tests.
PR #6787 changed the behavior of the `--EXTENSIONS--` section, so that
not yet loaded extensions are dynamically loaded if possible. However,
when no tests are specified for the runner, only tests for already
loaded extensions are run, what defeats the purpose of the improvement
of the `--EXTENSIONS--` behavior. We cater to that by detecting
loadable extensions, and also run their tests.
This additional internal fiber API creates and manipulates a Fiber object, allowing any internal function to start, resume, or suspend a fiber. The existing zend_fiber_context API allows custom C-based fiber creation using the bundled switching context, but does not interact with the PHP VM. This API behaves the same as calling Fiber object methods from user code, switching EGs, and triggering the fiber switch observer. In general, the Fiber object methods call these new API methods.
This version of libxml introduced quite a few changes. Most of
them are differences in error reporting, while some also change
behavior, e.g. null bytes are no longer supported and xinclude
recursion is limited.
Closes GH-7030. Closes GH-7046.
Co-authored-by: Nikita Popov <nikic@php.net>
If ZTS is enabled, this can cause cwd_globals_ctor() to be called
multiple times, each with a freshly allocated virtual_cwd_globals
instance. At shutdown time however, cwd_globals_dtor() will call
realpath_cache_clean(), which then possibly cleans up the same
realpath_cache instance more than once. Using AddressSanitzer, this
shows up as a heap use-after-free.
To avoid this, add a helper function to do the actual work on one
instance of a realpath_cache, and call it both from cwd_globals_dtor()
and realpath_cache_clean(). The former uses the virtual_cwd_globals
parameter passed in via the destructor, the latter uses the CWDG()
macro.
The required symbol name seems to be very stable, so we can hardcode
a fallback. This avoids many false positives when PHP is not built
against valgrind.
zend_throw_error() and zend_error() are variadic functions. In Linux,
arguments are passed in registers, whereas they must be put on stack in
Macos [1][2].
In this patch:
1. preprocessor macro "__APPLE__" is used to distinguish the OS.
2. the third argument "CARG3" is the only variadic argument and put on
stack.
3. the invocation of zend_error() is converted back to regular call in
zend_jit_undefined_offset_stub() and zend_jit_undefined_index_stub().
With this patch, all ~4k test cases can pass for "nonZTS+CALL"
functional/tracing JIT with Macos Apple Silicion.
[1]
https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms#Overview
[2] https://github.com/below/HelloSilicon#listing-9-1
Change-Id: I49bc3233fc253ad3e77e8664464ff3e830dcd183
While the specified restriction was checked, the #[Attribute]
attribute did not specify the flags parameter, so that Reflection
returned incorrect information.
In particular, Attribute itself has a CLASS target, not an ALL
target.
This results in an assertion failure when running under -e,
because there is an additional EXT_STMT + NOP before the FREE.
I don't think there's a strong reason to handle UNREACHABLE_FREE
specially here (it's only important that we *do* handle it), so
I'm dropping the code rather than adjusting it to scan over
certain opcodes.
These tests fail with the following diff:
========DIFF========
date: Tuesday, July 7, 2009 8:41:13 PM EDT
002+ msgf: Wednesday, July 8, 2009 12:41:13 AM utc
002- msgf: Tuesday, July 7, 2009 8:41:13 PM usnyc
========DONE========
FAIL Bug #58756: w.r.t MessageFormatter [ext/intl/tests/bug58756_MessageFormatter_variant2.phpt]
========DIFF========
001+ quinta-feira, 17 de maio de 2012 4:35:36 da tarde utc
001- quinta-feira, 17 de maio de 2012 5:35:36 da tarde ptlis
========DONE========
FAIL MessageFormat accepts IntlCalendar args [ext/intl/tests/msgfmt_format_intlcalendar_variant4.phpt]
That is, it appears that the timezone does not get applied. I don't
know why this happens, and don't want to spend more time trying to
figure this out.
* PHP-8.0:
Fix register save/restore around calls. (This fixes ext/opcache/tests/jit/fetch_dim_rw_001.phpt with -d opcache.jit=1202 without PROFITABILITY_CHECKS)
While resolving the path, the last step will reduce it down to ""
(an empty string) and realpath() will resolve this to getcwd().
If open_basedir contains the CWD, then that means open_basedir
will be bypassed for paths that don't have any components that
exist (if one of the components exists, then we abort the realpath
loop at that point).
Closes GH-7015.
1. Generate exit table at the end of each trace (list of unconditional branches for all side exit)
2. Jump to this table if conditional branch to side exit can't be performed because of limited jump distance (+/-1MB) (avoid extra veneers for side exits).
3. During trace linking, update targets of conditional branches to this exit table, if target trace can't be reachd because of limited jump distance (+/-1MB)
If the connection times out rather than being refused outright,
don't wait the full 60 seconds that are used by default. If this
happens, the error message will also be different.
Now that we properly dereference references of the superglobals. we
also need to dereference contained references to avoid to string
conversion.
Closes GH-7014.
We need to set the preload autoloader even if preloading happened
in a different process. accel_preload() will only run in one.
In particular, this fixes the behavior if preload_user is used.
In the meantime, additional tests for ftp:// with opendir() have
been added, which tests actual behavior against a server. This one
just uses localhost (without running a server), which can make the
test very slow on systems that don't refuse the connection outright.
The previously used error does not occur on FreeBSD and instead
causes the operation to time out.
Also clean up the test a bit. And make sure it actually tests
something -- the point is that socket_last_error() should be 0
after socket_clear_error(). It's kind of pointless if the test
uses a wildcard for the final result instead.
Now GDB correctly shows backtraces that involves JIT-ed code for
functional/tracing JIT, HYBRID/CALL VM, x86/AArch64 CPU.
(opcache.jit_debug=0x100 should be set).
Now GDB correctly shows backtraces that involves JIT-ed code for
functional/tracing JIT, HYBRID/CALL VM, x86/AArch64 CPU.
(opcache.jit_debug=0x100 should be set).
The object hash is not particularly useful (anymore) and just
clutters the output. It encodes the same information as the
object ID, which is already part of the output.
"(struct.._offset+offsetof(zend_..struct, field))" might exceed the
range of ADD_SUB_IMM, leading to DynASM error DASM_S_RANGE_I for
ZTS+CALL test variant.
Using macro ADD_SUB_64_WITH_CONST_32 would fix it.
Change-Id: I3233cefbcd1ddea16e7de6725c2dc5ff43373916
As suggested by Dmitry, macro CMP_32_WITH_CONST is not necessary because
Z_TYPE_P(val) is of uchar type and it can be encoded as the imm field of
'cmp' instruction directly.
Change-Id: Icb8f9ee847b9a08cb1e9127e7faf6c89d1431922
Macro GET_ZVAL_LVAL is optimized in commit 6e3443d, using 'reg' as the
temporary register. However, the assertion would fail for "jit=1205".
This patch reverts this optimization for GET_ZVAL_LVAL.
Change-Id: Iecb134fa9438ef993e6393974caa5710657a462e
For macro "SAFE_MEM_ACC_WITH_UOFFSET* ldr/ldrb", the target register can
be used as temporary register to load the offset if needed. Hence one
temporary register is saved for macros LONG_CMP, GET_ZVAL_LVAL,
CMP_ZVAL_TYPE, IF_ZVAL_TYPE and IF_NOT_ZVAL_TYPE.
Change-Id: I43f35186cec9a4157bc73b1cae6643aa59140de2
Offset "Z_OFFSET(op1_addr)" might be loaded into temporary registers for
two times, i.e. by these two SAFE_MEM_ACC_WITH_UOFFSET respectively.
This patch removes this case.
Change-Id: Ib1fde9c9a39f6b5f1ca322dc27b0a9d9d51fbef0
Macros PUSH_BASE_ADDR, PUSH_ADDR_ZTS, GET_ZVAL_W2, SET_ZVAL_W2,
IS_32BIT, PUSH_ADDR and PUSH_ZVAL_ADDR are 32-bit platform dependent in
the x86 implementation, hence they are derepcated in AArch64.
Macro ADDR_OP1 is only used by PASH_* macros, and it is deprecated in
AArch64.
Macros MEM_OP3_3 and AVX_OP are platform dependent and they are not used
in AArch64.
Macro LONG_OP_WITH_32BIT_CONST is replaced with LONG_ADD_SUB_WITH_IMM.
Macro DOUBLE_OP is not used because we use temporary FP register to load
the value from address firstly and then conduct FP math operations with
the help of DOUBLE_MATH_REG.
Macro LONG_MUL is deprecated. MUL between two LONG values is implemented
in a different way from ADD or SUB mainly because extra temporary
register is needed in order to detect integer overflow. See function
zend_jit_math_long_long().
Change-Id: I249f28820aaa86c7638ecf3eee08f8750a232397
Macros CMP_*_WITH_CONST were introduced in [1]. We revisit all the uses
of 'cmp' instructions with constants, and apply these macros if
possible.
[1] https://github.com/php/php-src/commit/66ba9af
Change-Id: Idd233cb9afba84dd106fcc5ee8bf5417d91989c7
Macros ADD_SUB_*_WITH_CONST* were introduced in [1], but it would be
more accurate to use ADD_SUB_IMM even though ADD_SUB_IMM == CMP_IMM.
Regarding the definition of ADD_SUB_IMM, the comment is updated since it
is used to guard 'subs' and adds' instructions as well.
[1] https://github.com/php/php-src/commit/0609b97
Change-Id: I102ebbf626a438da061f6c01b01029807de1ff14
Currently temporary register TMP3 is used in particular for ZTS mode.
Hence it would be better not to use it in other sites except ZTS.
In this patch, the value in TMP1, i.e. "REG1 + IP", might be clobbered
and should be calcauted again.
Change-Id: I17434d4f4b8d6f1e71f49eadb42e06db975d7bbe
- use cbz/cbnz for comparison with zero
- use xzr/wzr instead of #0 in mov/cmp
- use "subs" imstead of "sub x1,...; cmp x1, xzr"
- avoid loading immediate value to register when it may be direclu used
as a source immendiate operand in ldr*/str*/add*/sub*/cmp
Add macro "BW_OP_32_WITH_CONST" for safe 32-bit logical operations with
constants.
Add macro "GET_LOW_8BITS" for 'and' operation with "0xff".
Change-Id: I57968069b43ac4c02d876e9290433a74b9d87742
The 'pimm' field of ldr/str instruction has different maximum values for
32-bit register and 64-bit register.
Change-Id: I14b4f1eedbcbb303696c22c1403b49d52426d390
This patch implements stub functions zend_jit_hybrid_hot_code_stub() and
zend_jit_hybrid_hot_counter_stub(), which is needed by HOT COUNTER mode,
i.e. 'opcache.jit=123x'.
Test case "defined_001.phpt" can pass now since HOT COUNTER mode is
required by this case.
Change-Id: I66efb07c75fc64723dc76eff533d024c868f8022
This patch implements function zend_jit_hybrid_profile_jit_stub(), which
is needed by ZEND_JIT_ON_PROF_REQUEST mode, i.e. 'opcache.jit=122x'.
Change-Id: Icb591c81675ffc077180229362882278ac8406c6
Similar to the AArch64 implementation, we compare "op" and "op << n >>
n" to detect integer overflow.
Note that instructions 'pushf' and 'popf' are used to save/restore the
flags set by 'cmp', otherwise, the follow-up 'shl' might change the
flags.
Note that test case "mul_004.phpt" can pass now for x86.
Change-Id: Ieee037f41dce298979b8a0964ca97598939c495c
We implement a simplified version. Every value with one single bit can
be encoded for logical instructions and we suppose this quick check can
cover a lot of common masks.
Besides, add macro TST_64_WITH_ONE since it's used often to test 64-bit
register with constant 1.
Change-Id: I850a6ac6acbe2d12f85180e407344580ee6fea61
As explained by Dmitry, 'ucomsd' in x86 sets 'p' flag, and it also
always sets 'z' and 'c' flags. [1]
Besides, remove one duplicate 'break'.
[1]. https://mudongliang.github.io/x86/html/file_module_x86_id_316.html
Change-Id: I767214c7ab8db31115801a3ae96b20320757899f
LONG MUL can be optimzied into left shift if either operand is a power
of two. Conditions "IS_SIGNED_32BIT()" and "is_power_of_two()" are used
to filter out invalid candidates. However, there exists one exception,
i.e. -2147483648(that is 0xffff,ffff,8000,0000). See the stand-alone
case[1].
Assume "a = 3; b = -2147483648;". The expected result of "a * b" is one
negative value. However, it would be optimized to "a << 31", which is
positive.
This trigger condition is refined.
1) For x86 implementation, another check for positive numbers is added.
Note that LONG type, i.e. zend_long, is defined as int32_t for x86 arch
and int64_t for x64 arch. This optimization only accepts values which
can be represented by int32_t type as default. See IS_SIGNED_32BIlT(),
2) For AArch64, we employ helper function zend_long_is_power_of_two()
since values of int64_t type are used.
Overflow detection for left shifting is added in this patch as well.
Note 1: bit helper functions are arch-independent and we move them into
zend_jit_internals.h.
Note 2: two test cases are added. Test case mul_003.phpt is used to
check the trigger condition and mul_004.phpt is designed to check
overflow detection.
Note 3: overflow detection for x86 is not implemented yet as I think
anotehr temporay register besides R0 is needed. Hence mul_004.phpt would
fail on x86 machine.
If we can use R1 as tmp_reg, the code can be updated as below.
```
| GET_ZVAL_LVAL result_reg, op1_addr
if (may_overflow) {
use_ovf_flag = 0;
/* Compare 'op' and '((op << n) >> n)' for overflow.
* Flag: jne -> overflow. je -> no overflow.
*/
tmp_reg = ZREG_R1
| mov Ra(tmp_reg), Ra(result_reg)
| shl Ra(tmp_reg), floor_log2(Z_LVAL_P(Z_ZV(op2_addr)))
| sar Ra(tmp_reg), floor_log2(Z_LVAL_P(Z_ZV(op2_addr)))
| cmp Ra(tmp_reg), Ra(result_reg)
}
| shl Ra(result_reg), floor_log2(Z_LVAL_P(Z_ZV(op2_addr)))
```
[1]. https://godbolt.org/z/1vKbfv8oG
Change-Id: Ie90e1d4e7c8b94a0c8f61386dfe650fa2c6879a1
Overflow detection for LONG MUL is added in this patch. Quite different
from 'subs' and 'adds' where overflow can be easily checked via the V
flags, LONG MUL wouldn't set the flags.
We use 'smulh' instruction to get the upper 64 bits of the 128-bit
result and check the top 65 bits to tell whether integer overflow
occurs. [1]
Note that LONG MUL can be substituted by 'adds' or 'lsl' in some cases.
Hence, flag 'use_mul' is introduced in order to select the proper
overflow check check instruction afterwards.
[1]
https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/detecting-overflow-from-mul
Change-Id: I67e8287e9044c2a96b188d4bf6674736713abfe9
Immediates in different lengths are used by AArch64 instructions.
Previous use of MAX_IMM12 is not accurate.
Note that immediate value can be optionally shifted and this mode is not
supported currently. We may want to support this in the near future.
1. Pre-allocated bytes are missing in function
zend_jit_assign_const_stub().
2. 'w' register should be used for macro SET_ZVAL_TYPE_INFO.
3. 'w' register should be used to load "num_args" in function
zend_jit_do_fcall().
4. Remove the local path name in test case recv_002.phpt
5. One option is disabled temporarily in [1] and several test cases
would fail, e.g. shift_right_003.phpt. I suppose new execution paths are
touched. We will support them in the near future.
[1] https://github.com/php/php-src/commit/63d673d
Opcodes FE_RESET_R and FE_FETCH_R are met for the first time. Updates in
funtions zend_jit_fe_reset() and zend_jit_fe_fetch() are made to support
them.
Besides, one new path is covered in function zend_jit_inc_dec() for
opcode POST_INC.
Opcodes for $test are:
BB0:
0000 #1.CV0($char_code) [rc1, rcn, any] = RECV 1
BB1:
0001 #2.T1 [rc1, ...] = BW_AND #1.CV0($char_code) ...
0002 #3.T2 [bool] RANGE[0..1] = BOOL_NOT #2.T1 [rc1, ...]
0003 #4.T1 [bool] RANGE[0..1] = IS_EQUAL #1.CV0($char_code) ...
0004 JMPZ #4.T1 [bool] RANGE[0..1] BB3
...
New path is covered in function zend_jit_long_math_helper() for opcode
BW_AND.
New path is covered in function zend_jit_bool_jmpznz() for opcode
BOOL_NOT.
Major changes lie in functions zend_jit_cmp(), zend_jit_cmp_slow() and
zend_jit_check_exception_undef_result() to support opocdes IS_EQUAL and
JMPZ.
Opcodes for $Test::method are:
BB0:
0000 #0.T0 [rcn, any] = FETCH_OBJ_R THIS string("prop")
0001 #1.T0 [bool] RANGE[0..1] = JMPZ_EX #0.T0 [rcn, any] BB3
BB1:
0002 #2.T1 [rcn, any] = FETCH_OBJ_R THIS string("prop")
0003 INIT_METHOD_CALL 0 #2.T1 [rcn, any] string("method2")
0004 #3.V1 [ref, rc1, rcn, any] = DO_FCALL
...
New path is covered in functions zend_jit_fetch_obj() and
zend_jit_zval_copy_deref() for FETCH_OBJ_R THIS opcode.
New path is covered in function zend_jit_init_method_call() for opcode
INIT_METHOD_CALL.
Major chagnes lie in function zend_jit_bool_jmpznz() to support opcode
JMPZ_EX.
Note that macro ZVAL_DTOR_FUNC is updated to remove the hard-coded use
of REG0.
Opcode SEND_VAR_EX used in $test and opcode ZEND_SEND_VAR_NO_REF_EX used
in $main cover two new branches in function zend_jit_send_var()
respectively.
The updates in function zend_jit_bool_jmpznz() are made to support
opcode JMPNZ_EX used in $test.
Opcodes for function $evaluate are:
BB0:
0000 ASSIGN_OBJ THIS string("evalParameters")
0001 OP_DATA array(...)
0002 INIT_NS_FCALL_BY_NAME 2 string("A\extract")
0003 CHECK_FUNC_ARG 1
0004 V1 = FETCH_OBJ_FUNC_ARG (ref) THIS string("evalParameters")
0005 SEND_FUNC_ARG V1 1
0006 T1 = FETCH_CONSTANT (unqualified-in-namespace) ...
0007 SEND_VAL_EX T1 2
0008 DO_FCALL_BY_NAME
...
Major changes are made in function zend_jit_fetch_constant() to support
FETCH_CONSTANT opcode.
Besdies, cold code is touched in functions zend_jit_check_func_arg() and
zend_jit_send_var() for opcodes CHECK_FUNC_ARG and SEND_FUNC_ARG
respectively.
Updates in function zend_jit_type_check() are made to support TYPE_CHECK
opcode for statement "is_array($type)".
New path is touched in function zend_jit_concat_helper() to support
opcode CONCAT for statement "$type ."ops"".
Besides, one new path is covered in function zend_jit_return() when
leaving.
Part of generated opcodes for $foo are:
...
BB1:
0002 INIT_FCALL 1 96 string("foo")
0003 #5.V1 [rcn, object (instanceof A)] = FETCH_THIS
0004 SEND_REF #5.V1 [rcn, object (instanceof A)] 1
0005 DO_UCALL
Updates in functions zend_jit_fetch_this() and zend_jit_load_this() are
made to support FETCH_THIS opcode.
One new path is covered in function zend_jit_send_ref() by SEND_REF
opcode.
Instructions 'cqo' + 'idiv' are used in x86 to conduct MOD operation.
Specific registers RDX:RAX are used.
We use instructions 'sdiv' + 'msub' to accomplish the equivalent
functionality[1], and there is no such contrain on registers.
Similary to left/right shift operations in the previous patches,
boundary values, i.e. the second operand is 0 or -1, are handled.
[1]
https://stackoverflow.com/questions/35351470/obtaining-remainder-using-single-aarch64-instruction
This patch supports SL opcode. The range of the second operand is
checked against 0 and 64. If the second operand is negative, exception
would be raised. If the second operand is >= 64, the result is 0.
Besides, new path in macro ZVAL_COPY_VLAUE is covered for RETURN opcode.
This test case is a big one. This patch mainly handles
smart_branch_opcode cases in function zend_jit_cmp_double_common().
Note that I failed to construct test cases to verify whether the missing
NaN check in x86 is buggy or not. One TODO is left to remind us when the
relevant code is touched.
The following opcodes would be generated for $foo:
0000 #2.CV0($test) [bool] RANGE[0..1] = RECV 1
0001 #3.CV1($x) [long] RANGE[MIN..MAX] = RECV 2
0002 JMPZ #2.CV0($test) [bool] RANGE[0..1] BB4
0003 #4.T2 [bool] ... = IS_SMALLER_OR_EQUAL int(1) #3.CV1($x) ...
0004 JMP BB5
...
The updates in function zend_jit_verify_arg_type() are made to support
RECV opcode.
The updates in function zend_jit_bool_jmpznz() are made to support JMPZ
opcode.
New path is covered in functions zend_jit_cmp() and
zend_jit_cmp_long_long() for IS_SMALLER_OR_EQUAL opcode.
'smart_branch_opcode' JMPZ is used in this test case. Similar to the
previous patch, I still didn't get why NaN check is missing for the
cases ">" and ">=". In our implementation, we add such checks.
Comparison between LONG and DOUBLE is (partially) supported in a similar
way to comparison between two LONG values. See the updates in function
zend_jit_cmp().
Key difference lies in handling NaN.
1. Instruction 'fcmp' is used to substitue 'ucomisd' in x86
implementation. Both of them raise invalid operation exception only when
either source operand is an SNaN.[1][2]
2. Parity flag is used in x86 to check whether either operand is NaN.[3]
I think this is QNaN case. As for AArch64, we use instruction 'bvs'.[4]
It's worthing noting that condition codes have different meanings for
floating-point comparions(e.g. 'fcmp')[4] compared to the
general-purpose comparisons(e.g. 'cmp').[5] For instance, 'b.hs' after
'fcmp' can check not only the cases "greater than, equal to" but also
the case "unordered"(that is NaN). We may simply treat it as a
combination of 'jae' and 'jp' in x86.
3. Instruction 'SETcc' is used in x86 for the case of ">=" or ">".
Note that flag "swap" is set in implementation, and it falls into cases
ZEND_IS_SMALLER or ZEND_IS_SMALLER_OR_EQUAL. We can use 'cset' in
AArch64.
However, it's weird that the NaN check is missing in x86. I suppose it
might be a bug. Take the case ">=" as an example. The two operands can
be either DOUBLE + LONG or DOUBLE + DOUBLE. See the relevant code where
flag "swap" is set(i.e. function zend_jit_cmp_double_long() and function
zend_jit_cmp_double_double()). For the case "NaN >= 1.0", the expected
result should be FALSE, however, JIT/x86 would produce TRUE due to the
following "setae al". Unfortunately I haven't constructed one test case
to trigger that.
In our implementation, we choose to follow the case of "<" or "<=", and
I believe our implementation is safe anyway..
4. Temporary FP register is also needed and we reserve v16. See the
updates in file zend_jit_arm64.h.
5. Macro SET_ZVAL_TYPE_INFO_FROM_REG is misused in function
zend_jit_zval_copy_deref(). The second argument should be 32-bit long
and we fix it.
Note that simple test cases involving NaN are tested locally. I believe
it would get deeper testing by cmp_003.phpt(we will support it later).
[1]
https://developer.arm.com/documentation/dui0204/f/vector-floating-point-programming/vfp-instructions/fcmp?lang=en
[2] https://www.felixcloutier.com/x86/ucomisd
[3] https://en.wikipedia.org/wiki/Parity_flag
[4]
https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/condition-codes-4-floating-point-comparisons-using-vfp
[5]
https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/condition-codes-1-condition-flags-and-codes
This test case is a big one. Major changes are:
1. statement "foo($obj->a)"
One new path is covered in function zend_jit_fetch_obj() for the
involved FETCH_OBJ_W opcode. See the update around label 5.
Opcode SEND_REF is used. The updates in function zend_jit_send_ref() are
made to support it. Note that macro FREE_OP is executed for the first
time. Temproray registers are passed since they are used inside. As a
result, its use sites are updated accordingly.
2. statement "$a = array()" in $foo2
One new path in function zend_jit_assign_to_variable() is covered.
3. statements involving variable $d in $bar
One new path in function zend_jit_fetch_obj() is covered. See the
updates around label 7.
Note that in macro EMALLOC, condition ZEND_DEBUG can be covered by DEBUG
build, i.e. "./configure --enable-debug".
One new path is covered inside function zend_jit_fetch_obj() due to the
use of FETCH_OBJ_R opcode. Note that function zend_jit_zval_copy_deref()
is invoked along this new path.
Updates in function zend_jit_free() are made to support FREE opcode.
Stub function zend_jit_leave_function_stub() is touched for the first
time.
Opcode ASSIGN_OBJ is generated for statement "$x->a = 1;" and one new
path in function zend_jit_assign_obj() is covered. Note that function
zend_jit_assign_to_variable_call() is invoked along this new path.
Besides, helper function zend_objects_store_del() is used as the dtor
for objects.
Opcode FETCH_DIM_RW is not touched before and the udpates in function
zend_jit_fetch_dim() and zend_jit_fetch_dimension_address_inner() are
made to support it.
Besides, one new path is covered in function zend_jit_return() when
leaving.
The opcodes for function $foo are:
0001 INIT_FCALL 1 96 string("var_dump")
0002 #2.T1 [null, long] = FETCH_DIM_R array(...) #1.CV0($n) [...]
0003 SEND_VAL #2.T1 [null, long] 1
0004 DO_ICALL
0005 RETURN null
Opcode FETCH_DIM_R is not touched before, and the updates in function
zend_jit_fetch_dim_read() are made to support it.
As different types of arguments are used for $foo, several cases in
function zend_jit_fetch_dimension_address_inner() are covered as well.
Besides, opcode DO_ICALL can reach one site of cold code in function
zend_jit_do_fcall().
1. For statement "$a->change($a = array("a" => range(1, 5)));", the
following opcodes will be generated:
0002 ASSIGN CV0($a) V1
0003 INIT_METHOD_CALL 1 CV0($a) string("change")
0004 INIT_NS_FCALL_BY_NAME 2 string("A\range")
0005 SEND_VAL_EX int(1) 1
0006 SEND_VAL_EX int(5) 2
0007 V1 = DO_FCALL_BY_NAME
The updates in function zend_jit_init_fcall(), zend_jit_send_val() and
zend_jit_do_fcall() are made to support INIT_NS_FCALL_BY_NAME,
SEND_VAL_EX and DO_FCALL_BY_NAME respectively.
2. For method $change(), opcode RECV is used to obtain the argument:
0000 #1.CV0($config) [rc1, rcn, array of [any, ref]] = RECV 1
Accordingly the updates in functions zend_jit_recv() and
zend_jit_verify_arg_type() are made.
3. For statement "array_keys($config["a"])", the following opcodes will
be generated:
0001 INIT_NS_FCALL_BY_NAME 1 string("A\array_keys")
0002 CHECK_FUNC_ARG 1
0003 #3.V1 [ref, rc1, rcn, any] = FETCH_DIM_FUNC_ARG #1.CV0($config)
... -> #2.CV0($config) [rc1, rcn, ...
0004 SEND_FUNC_ARG #3.V1 [ref, rc1, rcn, any] 1
0005 #4.V1 [ref, rc1, rcn, any] = DO_FCALL_BY_NAME
CHECK_FUNC_ARG and SEND_FUNC_ARG are not supported before. See the
updates in functions zend_jit_check_func_arg() and zend_jit_send_var().
Besides, a new path is covered in macro OBJ_RELEASE when leaving.
1. For statement "echo $a->test()", opcode INIT_METHOD_CALL is
involved. The updates in function zend_jit_init_method_call() and
zend_jit_push_call_frame() are made to support it.
2. The updates in function zend_jit_leave_func() are made to support the
RETURN opcode used in functions $closure and $test.
3. The updates in function zend_jit_assign_to_variable() are used to
support statement "$x = $arr".
4. The updates in function zend_jit_fetch_dimension_address_inner() and
zend_jit_simple_assign() are made to support statement "$x['a'] =
$closure()", where opcode ASSIGN_DIM is involved.
For function Foo(), the original handlers would be invoked for the first
two statements. And the third statement "$a = 42", where ASSIGN opcode
is involved, covers the cold code in function
zend_jit_assign_to_variable().
For function $main(), statement "var_dump(Foo::$prop);" covers a new
path in function zend_ jit_send_val() for SEND_VAL opcode.
Besides, another 2 test cases, i.e. fetch_dim_r_003.phpt and
fetch_dim_r_004.phpt, would pass as well with this patch.
There are 6 user function calls in this test cases. The first 3
functions, i.e. foo(), foo1() and foo2(), can be supported already. In
this patch, we mainly focus on foo3(). Note that based on my test, once
foo3() gets supported, the remaining functions foo4() and foo5() can
pass as well.
Regarding function foo3(), we mainly focus on statement "$array = new
ArrayObject();", and the following two opcodes are involved.
0009 V2 = NEW 0 string("ArrayObject")
0010 DO_FCALL
Accordingly, functions zend_jit_handler(), zend_jit_cond_jmp() and
zend_jit_do_fcall() are invoked to generate the machine code. See the
handling process for case ZEND_NEW at file zend_jit.c. Hence, major
changes in this patch are made to support this statement.
Note that the updates at line 4840 in function zend_jit_do_fcall() are
made to support the later internal function call, i.e. var_dump().
Note that another test "noval_001.phpt" would pass with this patch as
well.
For statement "$a = new stdClass;", opcode NEW is used and JIT would
invoke the original handler at runtime.
Our major changes are made to support statements "$a->a=1;" and
"$a->b=2;" where opcode ASSIGN_OBJ are used.
Major changes are:
1. Support opcode FETCH_DIM_W for "$arr[0][0] = $ref;" in the loop. See
the updates in function zend_jit_fetch_dim().
2. Spill the registers and store the values into memory. See the updates
in function zend_jit_spill_store(). This is done for Phi function.
3. Invoke function zend_array_destory() as dtor for arrays. This is done
by zend_jit_free_cv() when leaving the function foo().
Major changes are made to support statement "$a[0] = $unref", where
opcode ASSIGN_DIM is involved.
Besides, one bug in macro GC_DELREF is fixed. The reference count would
be further checked after decreasing in macro ZVAL_PTR_DTOR, hence,
instruction "subs" should be used to set the flags. After fixing this
bug, external function zend_jit_array_free() is used as the dtor for the
array "$a".
Support the case where arguments might be reference.
Besides, another two test cases, assign_019.phpt and assign_032.phpt,
would pass as well with this patch.
Following the previous patch, we continue to support failed JIT test
cases involving reference.
In assign_010.phpt, major changes are done to support the assignment "$a
= $b" where "$b" is a reference.
Honestly speaking, I didn't fully understand the syntax here but rather
to translate the x86 implementation into AArch64.
Besides, test case assign_011.phpt would pass as well with this patch.
Reference is involved in this test case, i.e. "$ref2 = & $ref1;".
1. Fix one bug in zend_do_fcall(). For each stack slot, the type
information gets initialized during the call frame allocation phase.
Opcode ZEND_ASSIGN_REF is associated to this statement. It's worth
noting that PHP JIT doesn't apply to this opcode actually. That means
the original handler(i.e. interpreter version) will be invoked at
runtime. Note that this mode works for a number of opcodes, not only
ZEND_ASSIGN_REF.
In the execution of original handler, the runtime type information of
$ref2 is accessed and this bug is triggered.
2. Support macros GET_Z_PTR and ZVAL_DEREF.
3. Cover new paths in function zend_jit_simple_assign() and macro
ZVAL_COPY_CONST.
1. one **hybrid** solution of register usage
After the discussion with Dmitry, we may want to propose one hybrid
solution of register usage.
1) Following the x86 implementation, we define REG0/1/2 to be the
scratch registers. Clever tricks are utilized in x86 implementation for
better register allocation. Note that we define REG0/1/2 as x8/9/10. One
reason is that R0 and FCARG1 should be distinguished.
2) Temporary registers are also reserved(i.e. they are excluded from the
candidates of register allocator), and they would be used due to the
different addressing modes in AArch64.
2. update the 'make clean' target.
3. remove the unnecessary AArch64 related macros in zend_jit_internal.h.
[ci skip]
Change-Id: I627157b88b2344530d705751eb7f73a223ed83e5
CustomizedGitHooks: yes
SUMMARY
We implemented a prototype of PHP JIT/arm64. Briefly speaking,
1. build system
Changes to the build system are made so that PHP JIT can be successfully
built and run on ARM-based machine.
Major change lies in file zend_jit_arm64.dasc, where the handler for
each opcode is generated into machine code. Note that this file is just
copied from zend_jit_x86.dasc and the *unimplemented* parts are
substitued with 'brk' instruction for future work.
2. registers
AArch64 registers are defined in file zend_jit_arm64.h. From our
perspectives, the register usage is quite different from the x86
implementation due to the different ABI, number of registers and
addressing modes.
We had many confusions on this part, and will discuss it in details in
the final section.
3. opcodes
Several opcodes are partially supported, including INIT_FCALL, DO_UCALL,
DO_ICALL, RETURN, ADD, PRE_INC, JMP, QM_ASSIGN, etc. Hence, simple use
scenarios such as user function call, loops, addition with integer and
floating point numbers can be supported.
18 micro test cases are added under 'ext/opcache/tests/jit/arm64/'. Note
that majority of these test cases are design for functional JIT, and
cases 'hot_func_*.phpt' and 'loop_002.phpt' can trigger tracing JIT.
4. test
Our local test environment is an ARM-based server with Ubuntu 20.04 and
GCC-10. Note that both HYBRID and CALL VM modes are supported. We
suggest running the JIT test cases using the following command. Out of
all 130 test cases, 66 cases can be passed currently.
```
$ make test TESTS='-d opcache.jit=1203 ext/opcache/tests/jit/'
```
DETAILS
1. I-cache flush
Instruction cache must be flushed for the JIT-ed code on AArch64. See
macro JIT_CACHE_FLUSH in file 'zend_jit_internal.h'.
2. Disassembler
Add initialization and jump target parse operations for AArch64 backed.
See the updates in file 'zend_jit_disasm.c'.
3. redzone
Enable redzone for AArch64. See the update in zend_vm_opcodes.h.
Redzone is designated to prevent 'vm_stack_data' from being optimized
out by compilers. It's worth noting that this 16-byte redzone might be
reused as temporary use(treated as extra stack space) for HYBRID mode.
4. stack space reservation
The definitions of HYBRID_SPAD, SPAD and NR_SPAD are a bit tricky for
x86/64.
In AArch64, HYBRID_SPAD and SPAD are both defined as 16. These 16 bytes
are pre-allocated for tempoerary usage along the exuection of JIT-ed
code. Take line 4185 in file zend_jit_arm64.dasc as an example. NR_SPAD
is defined as 48, out of which 32 bytes to save FP/IP/LR registers.
Note that we choose to always reserve HYBRID_SPAD bytes in HYBRID mode,
no matter whether redzone is used or not, for the sake of safety.
5. stack alignment
In AArch64 the stack pointer should be 16-byte aligned. Since shadow
stack is used for JIT, it's easy to guarantee the stack alignment, via
simply moving SP with an offset like 16 or a multiple of 16. That's why
NR_SPAD is defined as 48 and we use 32 of them to save FP/IP/LR
registers which only occupies 24 bytes.
6. global registers
x27 and x28 are reserved as global registers. See the updates in file
zend_jit_vm_helpers.c
7. function prologue for CALL mode
Two callee-saved registers x27 and x28 should saved in function
zend_jit_prologue() in file zend_jit_arm64.dasc. Besides the LR, i.e.
x30, should also be saved since runtime C helper functions(such as
zend_jit_find_func_helper) might be invoked along the execution of
JIT-ed code.
8. regset
Minor changes are done to regset operations particularly for AArch64.
See the updates in file zend_jit_internal.h.
REGISTER USAGE
In this section, we will first talk about our understanding on register
usage and then demonstrate our design.
1. Register usage for HYBRID/CALL modes
Registers are used similarly between HYBRID mode and CALL mode.
One difference is how FP and IP are saved. In HYBRID mode, they are
assigned to global registers, while in CALL mode they are saved/restored
on the VM stack explicitly in prologue/epilogue.
The other difference is that LR register should also be saved/restored
in CALL mode since JIT-ed code are invoked as normal functions.
2. Register usage for functional/tracing JIT
The way registers are used differs a lot between functional JIT and
tracing JIT.
For functional JIT, runtime C code (e.g. helper functions) would be
invoked along the execution of JIT-ed code. As the operands for *most*
opcodes are accessed via the stack slot, i.e. FP + offset. Hence there
is no need to save/restore local(caller-saved) registers before/after
invoking runtime C code.
Exception lies in Phi node and registers might be allocated for these
nodes. Currently I don't fully understand the reason, why registers are
allocated for Phi functions, because I suppose for different versions of
SSA variables at the Phi function, their postions on the stack slot
should be identical(in other words, access via the stack slot is enough
and there is no need to allocate registers).
For tracing JIT, runtime information are recorded for traces(before the
JIT compilation), and the data types and control flows are concrete as
well. Hence it's would be faster to conduct operations and computations
via registers rather than stack slots(as functional JIT does) for these
collected hot paths. Besides, runtime C code can be invoked for tracing
JIT, however this only happends for deoptimization and all registers are
saved to stack in advance.
3. Candidates for register allocator
1) opcode candidates
Function zend_jit_opline_supports_reg() determines the candidate opcodes
which can use CPU registers.
2) register candidates
Registers in set "ZEND_REGSET_FP + ZEND_REGSET_GP - ZEND_REGSET_FIXED -
ZEND_REGSET_PRESERVED" are available for register allocator.
Note that registers from ZEND_REGSET_FIXED are reserved for special
purpose, such as the stack pointer, and they are excluded from register
allocation process.
Note that registers from ZEND_REGSET_PRESERVED are callee-saved based on
the ABI and it's safe to not use them either.
4. Temporary registers
Temporary registers are needed by some opcodes to save intermediate
computation results.
1) Functions zend_jit_get_def_scratch_regset() and
zend_jit_get_scratch_regset() return which registers might be clobbered
by some opcodes. Hence register allocator would spill these scratch
registers if necessary when encountering these opcodes.
2) Macro ZEND_REGSET_LOW_PRIORITY denotes a set of registers which would
be allocated with low priority, and these registers can be used as
temporary usage to avoid conflicts to its best.
5. Compared to the x86 implementation, in JIT/arm64
1) Called-saved FP registers are included into ZEND_REGSET_PRESERVED for
AArch64.
2) We follow the logic of function zend_jit_opline_supports_reg().
3) We reserve 4 GPRs and 2 FPRs out from register allocator and use them
as temporary registers in particular. Note that these 6 registers are
included in set ZEND_REGSET_FIXED.
Since they are reserved, may-clobbered registers can be removed for most
opcodes except for function calls. Besides, low-priority registers are
defined as empty since all candidate registers are of the same priority.
See the updates in function zend_jit_get_scratch_regset() and macro
ZEND_REGSET_LOW_PRIORITY.
6. Why we reserve registers for temporary usage?
1) Addressing mode in AArch64 needs more temporary registers.
The addressing mode is different from x86 and tempory registers might be
*always* needed for most opcodes. For instance, an immediate must be
first moved into one register before storing into memory in AArch64,
whereas in x86 this immediate can be stored directly.
2) There are more registers in AArch64.
Compared to the solution in JIT/x86(that is, temporary registers are
reserved on demand, i.e. different registers for different opcodes under
different conditions), our solution seems a coarse-granularity and
brute-force solution, and the execution performance might be downgraded
to some extent since the number of candidate registers used for
allocation becomes less.
We suppose the performance loss might be acceptable since there are more
registers in AArch64.
3) Based on my understanding, scratch registers defined in x86 are
excluded from candidates for register allocator with *low possibility*,
and it can still allocate these registers. Special handling should be
conducted, such as checking 'reg != ZREG_R0'.
Hence, as we see it, it's simpler to reserve some temporary registers
exclusively. See the updates in function zend_jit_math_long_long() for
instance. TMP1 can be used directly without checking.
Co-Developed-by: Nick Gasson <Nick.Gasson@arm.com>
The exception for null default values here exists to keep compatibility
with PHP < 7.1 where "Foo $bar = null" was the canonical way to create
a nullable parameter. If the parameter is actually "?Foo $bar = null",
then clearly compatibility with PHP < 7.1 is not a concern, and we
can throw a deprecation notice.
There was a loophole here when it came to usage with named arguments,
which was not intended. Close the loophole thoroughly by actually
dropping the default value from the signature entirely. The default
is still used to make the type nullable, but not for anything else.
This makes debug_print_backtrace() use the same formatting as exception
backtraces. The only difference is that the final #{main} is omitted,
because it wouldn't make sense for limited backtraces, and wasn't there
previously either.
Before the zval -> zend_object migration, this code used macros
like FORMATTER_METHOD_FETCH_OBJECT_NO_CHECK, which internally
clear the error. Now that they are no longer used, we need to
manually clear the error.
Literal compaction was incorrectly assuming that literals with
the same base literal and the same number of related literals
would be equal. Maybe that was the case historically, but at
least it isn't true in PHP 8, where FETCH_CONSTANT and INIT_METHOD
have distinct literals at the second position.
Fix this by making the cache key a concatenation of all literals,
rather than just the base literal. We still distinguish the number
of related literals based on a bias added to the string hash.
The check for the number of bound parameters was only executed if
at least one was bound. We should also error if nothing was bound.
With mysqlnd, mysqlnd itself ended up emitting an error, but with
libmysqlclient this error condition would not be detected.
Because php supports doc comments on class constants, I believe it would also
make sense to support them on enum cases.
I don't have strong opinions about whether attributes should be moved to be the
last element or whether the doc comment should go after the attribute,
but the ast will likely change again before php 8.1 is stable.
So far, all attributes are the last ast child node.
I didn't notice that doc comments weren't implemented due to
https://github.com/php/php-src/pull/6489 being a large change.
https://wiki.php.net/rfc/enumerations
did not mention whether or not doc comments were meant to be supported
Instead of manually implementing this, use the standard mechanism.
This has minor behavior changes (e.g. doing an isset() will now
return false instead of throwing) which are more in line with
typical behavior.
This PR replaces the bundled libgd includes from #include <foo.h> with
#include "foo.h" for gd-related headers to avoid including headers that
may be available in system directories instead of the expected local
headers.
Closes GH-6975.
To fix https://bugs.php.net/bug.php?id=77372 and improve support of `<input type="file" name="files" multiple webkitdirectory>` I introduced another item to the `$_FILES` array called `full_path`, containing the full filename, as supplied by the user-agent.
Co-authored-by: Björn Tantau <bjoern@bjoern-tantau.de>
Create name using either zend_init_string_fast (no repetitions) or by concatenating the name with zend_print_ulong_to_buf. This is much more efficient than using snprintf. We also avoid repeated strlen() calculations.
The tests leak memory, likely due to a slightly unclean shutdown of
libfbclient. To avoid failing CI, we disable LSan for all the tests.
Closes GH-6966.
To achieve this we need to introduce a new wrapper function with
dummy arguments which calls pcntl_signal_dispatch() to respect
the function pointer signature for a tick function.
If `SQLDescribeParam()` fails for a parameter, we must not assume
`SQL_LONGVARCHAR` for any param which is not `PDO_PARAM_LOB`. At least
mapping `PDO_PARAM_INT` to `SQL_INTEGER` should be safe, and not
introduce a BC break.
Closes GH-6973.
This fixes a number of test failures on low optimization levels.
(e.g. use after free in Zend/tests/bug54268.phpt and tests/classes/protected_001.phpt with opcache.jit=1204)
compact() is documented (https://www.php.net/manual/en/function.compact) as a variadic function accepting parameters which are strings or arrays of strings referencing defined symbols.
In actuality, passing nonsense parameters e.g. compact(true, 42) merely returns an empty array. I propose throwing a warning in these cases, to prevent silent bugs.
[namespaces in bundled extensions](https://wiki.php.net/rfc/namespaces_in_bundled_extensions) changes for PSpell.
- Rename `PSpell` class to `\PSpell\Dictionary`
- Rename `PSpellConfig` class to `\PSpell\Config`
- Add entries to `UPGRADING` file.
Related: bd12c94f
This is a re-application of the original match against master.
The patch was originally applied to master, then reverted from
there, incorrectly applied to PHP-8.0, reverted from there due
to ABI break, and now lands on master again. We can only hope
that it does not get reverted again ;)
Use FCI/FCC structure instead of custom implementation which does the same.
This also fixes the "bug" which prevented static methods from being shutdown functions.
Closes GH-5829
Co-authored-by: Aaron Piotrowski <aaron@trowski.com>
mixed should be behaving the same way as no type here, and not
require X to be autoloaded. Everything apart from "void" is trivially
covariant to "mixed".
* PHP-8.0:
Revert "Remove no longer used "log_errors_max_len" ini directive (#6838)"
Add entries for log_errors_max_len INI directive removal
Remove no longer used "log_errors_max_len" ini directive (#6838)
This removes switching to main for fatal errors in fibers in favor of catching any zend_bailout in a fiber and calling zend_bailout again after switching to the previous fiber or {main}.
We need to cater to references, when traversing the files to extract.
While we're at it, we move the `zval_file` declaration into a narrower
scope.
Closes GH-6959.
* Fix return type of php_user_filter::onCreate()
* Fix return type of mysqli::init()
* Fix return type of UConverter::fromUCallback() and UConverter::toUCallback()
While the typed property ensures that the value is a string,
we should make sure that we handle an unset property gracefully.
Do this by throwing the same error we would normally throw if
you access an uninitializde typed property.
This variable can indeed be uninitialized when passed to convert_cp,
though it will not actually be used if uninitialized. This is
still UB though, so let's fix it.
zend_string_dup() can return the original string if it is interned.
Of course, the string can't be interned here, but GCC doesn't know
that and throws a warning in release builds.
Replace zend_string_dup() (which is a bad API anyway) with a direct
call to zend_string_init(), which makes it more obvious that the
original alloca'd string cannot be reused.
open_mode was initialized to "wb", but the length set only to 1.
The effect of this was that the stream is opened using "wb", but
we only report "w".
Fix the length to report the actually used open_mode.
Followed by the bundled extension namespace RFC, #6925 updated the `IMAPConnection` class to `IMAP\Connection`.
This updates the UPGRADING file to reflect that change.
This is what we normally do for fatal errors. The reason why this
became necessary now, is that a bailout can switch from a fiber
back to the main stack. In that case we do not want to try
destroying the fiber.
Fixes oss-fuzz #33917.
This is meant to test against certain fixed responses of Firebird
servers. For now we add just a most basic test which verifies a
connection attempt.
Closes GH-6940.
1. Update: http://www.php.net/license/3_01.txt to https, as there is anyway server header "Location:" to https.
2. Update few license 3.0 to 3.01 as 3.0 states "php 5.1.1, 4.1.1, and earlier".
3. In some license comments is "at through the world-wide-web" while most is without "at", so deleted.
4. fixed indentation in some files before |
Make sure that the previous opline is part of the same block,
otherwise it may be non-dominating.
The test case does not fail on PHP-7.4, but I think the general
problem can appear on 7.4 as well, so I'm applying the patch to
that branch.
Checking EG(current_exectue_data) throws into the previous fiber instead of triggering a fatal error during shutdown. A fatal error is triggered only if the throwing destroyed fiber was resumed from {main}.
The ?array $ctorArgs = null parameter is changed to array $constructorArgs = [], and an additional memory leak revealed by the new test case is fixed.
Closes GH-6937
Co-Authored-By: Nikita Popov <nikita.ppv@gmail.com>
In the added test case, the Closure ends up being freed before
the generator during GC.
This patch closes the generator (and thus releases the held
closure / execute_data) already during dtor_obj, which will avoid
ordering issues in free_obj. dtor_obj is not always called, but
if it isn't, then we also won't run GC and will free_obj in
reverse construction order.
Fixes oss-fuzz #33947.
Only reset the uninitialized property flag once the type check
has succeeded. Previously the property was treated as unset rather
than uninitialized after a failed assignment.
Noticed this edge-case while working on accessors...
First we need to properly clear the `inbuf`, what is an amendment to
commit d2881adcbc[1].
Then we need to report `php_pollfd_for_ms()` failures right away; just
setting `errno` does not really help, since at least in some cases it
would have been overwritten before we actually could check it. We use
`php_socket_strerror()` to get a proper error message, and define
`ETIMEDOUT` to the proper value on Windows; otherwise we catch the
definition in errno.h, which is not compatible with WinSock. The
proper solution for this issue would likely be to include something
like ext/sockets/windows_common.h.
Finally, we ensure that we only report warnings using `inbuf`, if it is
not empty.
[1] <http://git.php.net/?p=php-src.git;a=commit;h=d2881adcbc9be60de7e7d45a3316b0e11b7eb1e8>.
Closes GH-6718.
We always run test steps on azure, even if a previous step failed, because we
don't want a failure in one configuration to prevent testing of other
configurations. However, if building PHP fails in the first place, then we
currently end up testing the system PHP binary. This is not only useless, but
will also result in a test runner hang and cause the job to time out.
Avoid this by using sapi/cli/php instead of the installed php as the test
binary. If we fail to build, then it will not exist, and testing will fail
immediately.
Previously an exception thrown during fiber destruction resulted in a fatal error, but that exception should be able to be caught (unless we’ve entered shutdown, then still use a fatal error so the error is not hidden).
Userland property infos are no longer duplicated since PHP 7.4, when we
stopped setting SHADOW flags on inherited private properties. Stop duplicating
internal property infos as well.
This requires switching class destruction to work in reverse order, as child
classes may be reusing structures from parent classes, and as such should be
destroyed first.
Since 3e6b447979 it is again possible to have
warnings (deprecations) during inheritance, and more such functionality is
likely in the future. This is a problem, because such warnings will only be
shown on the first request if the opcache inheritance cache is used. This
currently causes test failures in --repeat builds.
Fix this by uplifting the error recording functionality from opcache to Zend,
and then using it to persist a warning trace in the inheritance cache, which
can then be used to replay the warnings on subsequent executions.
This is needed by both fibers and opcache (and GH-6903 also uses it),
so make it a common structure that can be used by any functionality
storing warnings/errors.
If Serializable is implemented, require that __serialize() and
__unserialize() are implemented as well, else issue a deprecation
warning.
Also deprecate use of PDO::FETCH_SERIALIZE.
RFC: https://wiki.php.net/rfc/phase_out_serializable
Closes GH-6494.
As we already have the called zend_function here, let's use the
name directly, rather than using the lcname of the call. This means
that aliases would be handled correctly automatically.
The ASSERT_CALLBACK value is not validated at all -- it's possible
to set it to an arbitrary value. As such, the function can also
return any value or type (even without outright abuse, the opcache
func info was wrong in that the return can be rcn, and the array
can be array_of_ref).
Default them to IntlDateFormatter:FULL, which has value 0, which
was what you would get if you passed null to these arguments in
weak typing mode. The documentation called this ICU's default
date/time type.
The `StrLen_or_IndPtr` parameter usually may be `SQL_NO_TOTAL`; we need
to cater to that possibility to avoid working with negative string
lengths and other issues. A noteable exemption are calls to
`SQLGetData()` which return `SQL_SUCCESS`; in that case `SQL_NO_TOTAL`
can not occur.
Closes GH-6809.
We can no longer export from git.php.net, and apparently exporting from
Github is not supported. We apply a quick fix to export from the local
clone by default, still leaving an option to export from some other
repo. This is, unfortunately, a minor BC break in a security release.
Co-authored-by: Remi Collet <remi@php.net>
Closes GH-6919.
Like `mail()` and `mb_send_mail()`, `imap_mail_compose()` must prevent
header injection. For maximum backward compatibility, we still allow
header folding for general headers, and still accept trailing line
breaks for address lists.
PDO drivers allow for it, and procedural ODBC has its own facility
for it, but PDO_ODBC doesn't. If a connection is severed (for example,
on IBM i, ending a databse job, or killing the network connection
elsewhere), a persistent connection could get stuck. This adapts the
procedural ODBC code to PDO for handling connection liveness, so PDO
can reconnect if needed.
A discussion about the method to check liveness is linked; this might
not be the best method, but it's what procedural ODBC uses, so it's
consistent.
Closes GH-6805.
This reverts commit 343d1c2b1f.
Also remove the incompatible test data for time being. The new line
handling will need an extra fix.
Signed-off-by: Anatol Belski <ab@php.net>
Integrating the upstream tests is important to ensure the quality of the
patch and to ensure the parity in the function.
Signed-off-by: Anatol Belski <ab@php.net>
PHPDBG won't parse opcodes correctly if they are more than two words
separated by underscores (as many opcodes, like ZEND_POST_INC, are).
Now opcodes that have more than one underscore (e.g. ZEND_POST_INC) are
parsed correctly.
Closes GH-6895.
The deprecation of passing null is thrown otherwise.
If the timeout is calculated conditionally before calling stream_select(), passing to avoid the deprecation seems unreasonable, example:
https://github.com/amphp/amp/blob/7d4bbc6e0b47c6bb39b6cce1a4b5942e0c5125fb/lib/Loop/NativeDriver.php#L286
Also enforce that if $seconds is null, then $microseconds should be null as well. However 0 is still accepted (with deprecation) for backwards compatibility.
If fdatasync is not available, use fsync instead.
For some reason, fdatasync is available when using clang from Xcode, even though no declaration is present in headers.
`php_stream_can_cast()` forwards to `_php_stream_cast()` with `ret` set
to `NULL`. `php_pgsql_fd_cast()` needs to cater to that, because
otherwise the stream would report that it is not castable.
This *might* fix https://bugs.php.net/73903.
Closes GH-6888.
* Prevents some unnecessary strlen() computation
* Use interned "NULL"
* Certain PGSQL_API functions now accept zend_string* instead of char*
Closes GH-6792
Firstly, we must not forget to set appropriate error codes for "manual"
checks in `virtual_file_ex()`.
Secondly, we must not call `php_error_docref2()` for warnings regarding
unary functions; thus, we introduce `php_win32_docref1_from_error()`.
Closes GH-6872.
The never type can be used to indicate that a function never
returns, for example because it always unwinds.
RFC: https://wiki.php.net/rfc/noreturn_type
Closes GH-6761.
Paragon Initiative Enterprises is aware of PHP applications that use sodium_compat's ParagonIE\Sodium\Core\XChaCha20 class directly for stream encryption.
Greater performance and security assurance is offered by exposing libsodium's crypto_stream_xchacha20 API to PHP users.
It's acceptable to only include this change in PHP 8.1+; the offending applications are more than welcome to either install ext/sodium from PECL or upgrade to 8.1 when it comes out later this year.
Ref: https://github.com/jedisct1/libsodium-php/pull/211
When we need to enable ASan/UBSan in PHP extension, we also need to enable it in PHP kernel. Providing these options makes it easier for us to enable ASan/UBSan when compiling PHP.
And we use --enable-address-sanitizer and --enable-undefined-sanitizer in azure-pipelines.yml instead of changing CFLAGS manually.
We also add compile flag check to MSan.
The function name should be kept if Closure was created from the function which is marked as ZEND_ACC_CALL_VIA_TRAMPOLINE, because it is not a one-time thing and it may be called multiple times.
Closes GH-6867.
If the script will be cached in SHM (!corrupted), then we cannot
allocate the static variables on the arena. Instead do the same
thing we do during normal persistence and allocate a map ptr slot.
- Treat it as error if multi-byte string or escape sequence is truncated
- Don't allow 'control' characters or escape sequences to appear in the middle
of a multi-byte char
As with ISO-2022-JP-KDDI, the main reference used to develop the tests was
the behavior of the existing code. It would have been better to have some
independent reference which we could cross-check our code against, but I
couldn't find one.
- Treat it as an error if a multi-byte character or escape sequence is truncated
- When converting other encodings to ISO-2022-JP-KDDI, don't swallow trailing
hash characters or digits
- Don't allow 'control' characters to appear in the middle of a multi-byte char
Note: I was not able to find any kind of official or even semi-official
specification for this legacy encoding. Therefore, the test suite for
ISO-2022-JP-KDDI is based largely on the behavior of the existing code.
Verifying the correctness of program code in this way is very questionable.
In a sense, all you are proving is that the code "does what it does". However,
the test suite will still expose any unintended _changes_ to behavior.
To detect errors in conversion from Unicode to another text encoding, each
mbstring conversion filter object maintains a count of 'bad' characters. After
a conversion operation finishes, this count is checked to see if there was any
error.
The problem with CP50220 was that mbstring used a chain of two conversion filter
objects. The 'bad character count' would be incremented on the second object in
the chain, but this didn't do anything, as only the count on the first such
object is ever checked.
Fix this by implementing the conversion using a single conversion filter object,
rather than a chain of two. This is possible because of the recent refactoring,
which pulled out the needed logic for CP50220 conversion into a helper function.
There's no need to dynamically allocate a struct to hold the 'mode' parameter;
just store it directly in `filt->opaque`. Some other things were also being done
in an unnecessarily roundabout way.
Also, the 'copy' function for CP50220 conversion filters was *both* broken
and unnecessary. Broken, because it malloc'd memory which was never freed by
anything. Unnecessary, because the point of the copy is so that various
algorithms can try running bytes through a conversion filter and see how many
output bytes or characters result, and then back out by restoring the filters
to their previous state. But here's the thing; CP50220 conversion filters don't
hold cached bytes, which is the main thing which would need to be restored to a
previous state.
This function pointer is only called when initializing the struct. After that
nothing is done with it. Therefore, there is no need to keep it in the struct.
This constructor function doesn't do anything different than the generic one.
There's no need to invoke it, either, when initializing a CP50220 conversion
filter.
As debug_print_backtrace() is not performance-critical, this
implements it by formatting the zend_fetch_backtrace() result.
This means there is only one place implementing the backtrace
construction logic, and they cannot go out of sync.
zend_fetch_backtrace() has much better test coverage, because
it is used by exceptions.
Closes GH-6869.
Now similar "fake" frames now materialized when fetching debug
backtraces. The patch also fixes few incorrect backtraces for generators
in *.phpt tests.
Handle this in the implementation of get_current_key of user_it,
so that the callers may assume that the key is not a reference.
Fixes oss-fuzz #33018.
shell_exec() can return null both when an error occurs or the program produces no output, or return false when popen failed, and treating null/false as an empty string has no effect on the behavior of phar here.
Now that the value is coerced to the correct type, we should be
treating this as a boolean, not an integer (treating it as an
integer was already incorrect before -- if people used the
property as documented they'd likely get garbage).
`buf` may contain NUL bytes, so we must not use `strcspn()` but rather
a binary safe variant. However, we also must not detect a stray CR as
line ending, and since we only need to check line endings at the end
of the buffer, we can nicely optimize.
Co-authored-by: Nikita Popov <nikita.ppv@gmail.com>
Closes GH-6836.
fsync is a straightforward wrapper around the same C function
(implemented on Windows API as _commit() with identical signature).
From the man pages:
fsync() transfers ("flushes") all modified in-core data of (i.e.,
modified buffer cache pages for) the file referred to by the file
descriptor fd to the disk device (or other permanent storage
device) so that all changed information can be retrieved even if
the system crashes or is rebooted. This includes writing through
or flushing a disk cache if present. The call blocks until the
device reports that the transfer has completed.
RFC: https://wiki.php.net/rfc/fsync_function
Closes GH-6650.
PDO implement half of this, but this functionality is generally
useful. Provide these as zend_u64_to_str and zend_i64_to_str to
complement zend_long_to_str.
Previously this just assumed that the value was of a certain type.
I'm doing this in a generic way that checks against the declared
property type -- the handler function can then assume the value
to be of the correct type.
This PR corrects misspellings identified by the check-spelling action.
The misspellings have been reported at jsoref@b6ba3e2#commitcomment-48946465
The action reports that the changes in this PR would make it happy: jsoref@602417c
Closes GH-6822.
We can use unwind_exit instead of the evil zend_bailout without breaking the original behavior in this way because the original zend_bailout will longjmp out of the executor, so the exception will never be caught and it always triggers the E_ERROR here.
This optimization is targeting cases when a SplPriorityQueue instance is used
exclusively with double or long priorities.
During the first insertion into an empty queue, the comparator is changed to
the specialized one if the priority of inserted inserted key is long or double.
During insertion to non-empty queue, comparator is swapped back to the generic
one on type conflict.
As a result code like following, where the weight field is always double or
int, runs almost twice as fast.
foreach ($items as $item) {
$pqueue->insert($item, -$item->weight);
if ($pqueue->count() > $size) {
$pqueue->extract();
}
}
Function info for curl_exec() incorrect specified that the
function cannot return true. This is already fixed in PHP 8,
as the func info entry was removed there.
As mentioned in bug #80949, if a transaction is implicitly
committed, then PDO may still issue a ROLLBACK when the PDO
object is destroyed, as the transaction is still marked as active
in PDO's (inaccurate) transaction emulation.
Make sure we use the connection transaction state also for that
final ROLLBACK. A caveat here is that the connection might have
been dropped already, which may be the case for some drivers if
construction fails. Make sure we don't crash in that case.
We use the proper type, and make sure that no overflow can occur by
using `safe_emalloc()` (we can assume that neither string length is
`SIZE_MAX`).
Closes GH-6845.
The actual name of this function is dir(), but ever since it was
introduced in PHP 3, its internal name was "getdir", leading to it
being mistaken for an alias. This went unnoticed until the switch
to stubs for generating arginfo, at which point getdir() became a
real but undocumented function.
Fixes bug #80914.
Closes GH-6855.
* can_ret_as_str_in_uni and is_possibly_blob is no longer used anywhere
* Removed mysqlnd_stmt_fetch_row_buffered header
* Removed unicode constants leftovers
This reverts commit 4440ac3ed6.
This fix wasn't correct, as we do retain the arena-allocation
for the file cache only case. This will need some other way to
reconcile both modes.
This one had objid_query->array_output = ( (st & SNMP_CMD_SET) ? false : true ); initially
which means that if it was equal to 0 it would be equal to true.
Use RETURN_* macros instead of RETVAL_* + return;
Use proper boolean types
Use zend_string instead of char* to prevent unnecessary strlen calculation
Closes GH-6846
During unserialization, static_variables and runtime_cache may be
allocated on the arena, and then later freed when the checkpoint
is released.
As we're only doing a single large allocation here, simply move
this to the normal allocator, so there is no interference with
other arena allocations.
Nowadays op_arrays always hold a ref to the function_name, even
if they are fully shared. As such, I don't believe that this code
is necessary anymore, and it interferes with the code that
releases function_names the appropriate number of times while
persisting.
As zend_update_parent_ce() only runs later, the parent reference
may still point to the original class entry rather than the
persisted one. Memory held by the original class entry may have
already been deallocated. Avoid use-after-free by explicitly
fetching the persisted parent CE.
When serialising object properties, they are oftet may override the
default values, however default values are most often scalars, interned
strings or immutable arrays.
This is generalization of idea, that was previously usesd for caching
resolution of class_entries in zend_type. Now very similar mechanizm is
used for general zend_string into zend_class_entry resolution.
Interned zend_string with IS_STR_CLASS_NAME_MAP_PTR GC_FLAG uses its
refcount to adress corresponding zend_class_entry cache slot.
The refcount keeps an offset to this slot from CG(map_ptr_base).
Flag may be checked by ZSTR_HAS_CE_CACHE(str), cache slot may be read by
ZSTR_GET_CE_CACHE(str) and set by ZSTR_SET_CE_CACHE(str, ce).
* PHP-8.0:
Fix DCE of FREE of COALESCE
Note that this is a non-trivial merge, as opt/coalesce.phpt
regresses with this change. I'll have to see if it can be
improved.
When encountering the following SSA graph:
BB1:
#2.T1 [string] = COALESCE #1.CV0($str) [null, string] BB2
BB2:
#5.T1 [string] = QM_ASSIGN string("")
BB3:
#7.X1 [string] = Phi(#2.X1 [string], #5.X1 [string])
FREE #7.T1 [string]
We would currently determine that #7, #5 are dead, and eliminate
the FREE and QM_ASSIGN. However, we cannot eliminate #2, as
COALESCE is also responsible for control flow.
Fix this my marking all non-CV phis as live to start with. This
can be relaxed to check the kind of the source instruction, but
I couldn't immediately come up with a case where it would be
useful.
I suspect this is the cause for our recent CI failures. Apparently,
on macos it is possible for getgroups() to return more than
NGROUPS_MAX groups. We avoid an EINVAL in that case by fetching
the exact number of groups in advance. This should work on both
macos and posix systems.
For rationale, see #6787
Extensions migrated in part 4:
* simplexml
* skeleton
* soap
* spl
* sqlite3
* sysvmsg
* sysvsem
* tidy - also removed a check for an ancient dependency version
When `SSL_read()` after `SSL_shutdown()` fails with `SSL_ERROR_SYSCALL`,
we should not warn about this, because it is likely caused by the peer
having closed the connection without having sent a close_notify
shutdown alert.
Signed-off-by: Christoph M. Becker <cmbecker69@gmx.de>
Closes GH-6803.
Some user opcode handler actually gets called when JIT is used, so do not disable JIT even if these user opcode handlers were registered, just ignore them.
ZEND_EXIT can help Swoole to detect exit/die operation and exit from the current coroutine and record exit_status correctly.
ZEND_*_SILENCE can help Swoole to switch EG(error_reporting) to keep the right behavior when using the error control operator.
So we ignore ZEND_EXIT, ZEND_BEGIN_SILENCE, ZEND_END_SILENCE in JIT check here.
Closes GH-6640.
I found this mainly to be a problem when working with threading
extensions. zbacktrace doesn't show the correct stack when switching
threads.
Closes GH-6814.
For rationale, see https://github.com/php/php-src/pull/6787
Make extension checks lowercase, add a special case for opcache
that has internal name not matching .so filename.
Extensions migrated in part 2:
* dom
* exif
* fileinfo
* ffi
This doesn't seem to serve a purpose anymore. Stats are expensive,
so doing an unnecessary stat just to short-circuit the zero size
case is rather dubious. It can also break with stream wrappers
that return inaccurate sizes (symfony/symfony#40574) and probably
can also break with stream filters.
Drop the special case and adjust code to make it more obvious that
it will still be handled correctly.
Closes GH-6807.
`xmlNewNs()` does not XML encode the passed `href`, so we need to do
that manually.
Closes GH-6804.
Signed-off-by: Christoph M. Becker <cmbecker69@gmx.de>
This makes the json encoding behavior the same as it was prior to the memory
optimizations added in f9f8c1c79c
(for objects with declared properties)
This is based on the code for the unoptimized case below the changes.
Buggy output prior to this commit:
```
{
"prop":"value"}
```
Correct output:
```
{
"prop": "value"
}
```
Closes GH-6811
This reverts commit 5b01c4863f.
The previous commit fixes the 32-bit issue. Maybe this one is also
needed, but let's see if any failures of this form turn up first.
We need to do that in case a user handler has been set. However, we
can't do that in `php_rinit_session_globals()` since that function is
called by PHP function `session_destroy()` too, but in that case we
don't want to reset PS(mod).
Closes GH-6795.
`session_set_save_handler()` may be called with callables instead of an
object; we need to cater to that as well.
We also extract a set_user_save_handler_ini() function to avoid code
duplication.
Closes GH-6796.
This handles the degenerate case where SCCP replaced the value in
the RETURN opcode with a constant, but the VERIFY_RETURN is still
there. We can still apply the same optimization, just don't need
to adjust the use list in this case.
The result is still sub-optimal in that a dead QM_ASSIGN is left
behind.
Currently, most skip checks are just for making sure an extension is
available. Even with recent addition of skip caching, this makes tests
needlessly slow:
* Checks for the same extension in its tests can have small differences
impacting cacheability.
* Even identical skip checks in two tests can still be executed twice if
they're run by different workers.
To remedy this, I'm repurposing the existing --EXTENSIONS-- section of
.phpt files to specify wjich extensions are required for current test to
run. Current behavior:
1) If the extension is already visible to PHP, all is good
2) If it isn't, assume it's present as a shared module and attempt to add
it via a command line parameter
3) If that works, all is good
4) If it doesn't, PHP fails with a cryptic error message trying to
execute the test itself
After this commit:
1) and 2) are kept unchanged
3) Check if shared extension file from 2) is actually present
4) Skip the test if it isn't
Other benefits include clear skip reasons (vs. sometimes none in many
current skip checks) and moving test information from code to metadata,
opening more opportunities for search and analysis.
Since --EXTENSIONS-- is barely ever used, this change poses no risk of
hidden failures.
As a demonstration of the new approach, this commit migrates one
extension to it. If merged, I will migrate other extensions in
subsequent PRs.
Closes GH-6787.
There is no need to require a (valid) save_handler to be set, when a
user handler is supposed to be set. We just have to make sure, that
no user handler is already set in this case.
Closes GH-6788.
Even if we don't know the exact method being called, include it
in the call graph with the is_prototype flag. In particular, we
can still make use of return types from prototype methods, as
PHP 8 makes LSP violations a hard error.
Most other places are adjusted to skip calls with !is_prototype.
Maybe some of them would be fine, but ignoring them is conservative.
Even if an explicit return type is given, we might still infer
a more narrow one based on return statements. We shouldn't
pessimize this just because a type has been declared.
We have separate flags for non-autoloading class fetches and
silent class fetches. There's no reason why NO_AUTOLOAD should
be special-cased to be implicitly silent.
It is not guaranteed, that the driver inserts only a single NUL byte at
the end of the buffer. Apparently, there is no way to find out the
actual data length in the buffer after calling `SQLGetData()`, so we
adjust after the next `SQLGetData()` call.
We also prevent PDO::ODBC_ATTR_ASSUME_UTF8 from fetching garbage, by
fetching all chunks with the same C type.
Closes GH-6716.
The compile context is shared between patterns, so we need to set
the character tables unconditionally in case we switched from
a non-C locale to the C locale.
Now that constants can contain objects (currently only enums),
we should destroy them before we free the object store, otherwise
there will be false positive leak reports.
This doesn't affect the fast_shutdown sequence.
The function accepts a bool since PHP 8.0, so it should also return
a bool to keep things consistent.
Furthermore a null return from this functions is not possible.
According to the DOM standard, elements may only contain element, text,
processing instruction and comment nodes[1]. It is also specified that
a HierarchyRequestError should be thrown if a document is to be
inserted[2]. We follow that standard, and prevent the use-after-free
this way.
[1] <https://dom.spec.whatwg.org/#node-trees>
[2] <https://dom.spec.whatwg.org/#mutation-algorithms>
Closes GH-6765.
static_variables should be treated the same way as all other
op_array components nowadays (only static_variables_ptr is
special). There's no need to persist/serialize it is separately
per shared op_array.
This moves a bunch of code outside of run_tests(), making it a bit
more manageable. Additionally, accessors provide better readability
than isset() and friends.
This is a minimal patch that moves the code but does not refactor
much. For the sake of reviewing experience, it does not involve
further refactoring which could include:
* Removing setSection()
* Fixing up the mess with hasSection() vs. sectionNotEmpty(), only
one of which is really needed.
* Moving more repetitive code into the new class.
All of this will be done with later commits.
Closes GH-6678.
If an internal class overrides read_property and declared property
types, make sure that the returned value matches the declared
type (in debug builds).
* @@session.old_passwords is present only in 5.6 and 5.7. Newer versions
already behave as if they have old_passwords=2.
* SET PASSWORD FOR ... = PASSWORD(...) syntax was removed in 8.0.
For all affected tests, versions that don't support newer SQL are
already explicitly excluded with version checks.
Closes GH-6729.
If nocache is used, then we should return an empty string result,
so the test doesn't get marked as borked.
Also limit to an occurrence at the start, just like other skip
tags.
If the SKIPIF section return nocache, don't cache the result. This
is used by tests that perform complex test set in conjunction with
skip checks, and need to be evaluated every time.
* PHP-8.0:
Fix ASAN warning (Zend/zend_string.h:165:2: runtime error: null pointer passed as argument 2 of memcpy(), which is declared to never be null)
If the ZEROFILL flag is set for a field, do not convert it into
an integer (text protocol) or convert it explicitly into a padded
string (binary protocol).
Because of the memcpy, compilers can't infer that ZSTR_LEN (i.e. class_name->len)
did not change, so they copy it out of memory into a register for the last two
accesses.
php_var_serialize_string already does something similar.
Closes GH-6734
Currently, dynamically declared functions and closures are inserted
into the function table under a runtime definition key, and then later
possibly renamed. When opcache is not used and a file containing a
closure is repeatedly included, this leads to a very large memory leak,
as the no longer needed closure declarations will never be freed
(https://bugs.php.net/bug.php?id=76982).
With this patch, dynamic functions are instead stored in a
dynamic_func_defs member on the op_array, which opcodes reference
by index. When the parent op_array is destroyed, the dynamic_func_defs
it contains are also destroyed (unless they are stilled used elsewhere,
e.g. because they have been bound, or are used by a live closure). This
resolves the fundamental part of the leak, though doesn't completely
fix it yet due to some arena allocations.
The main non-obvious change here is to static variable handling:
We can't destroy static_variables_ptr in destroy_op_array, as e.g.
that would clear the static variables in a dynamic function when
the op_array containing it is destroyed. Static variable destruction
is separated out for this reason (we already do static variable
destruction separately for normal functions, so we only need to
handle main scripts).
Closes GH-5595.
Valgrind 3.3.0 was released in 2007, not even RHEL has crap older than
this. It could be argued that 3.8.0, released in 2012, could be a safe
cutoff too.
Closes GH-6728.
We already use <= for IS_SERIALIZED(), but the same general
problem can also occur for IS_UNSERIALIZED(). We don't seem to
hit this in practice prior to GH-5595 though.
Currently every --SKIPIF-- section in every test file results in 1
extra execution of PHP, every --EXTENSIONS-- section - in 2 executions.
This is quite wasteful, as skip checking code is extremely repetitive
and extensions are fixed for every binary/ini/command parameters
combination.
This patch adds caching to all such checks.
On my machine, the gains are quite noticeable: 36s instead of 43s
with -j16, 292s instead of 337s without concurrency. Cache stats are
3780 hits, 1247 misses in the latter case. In the future, tests could
be adjusted to have more uniform skip checks to improve performance even
more.
Closes GH-6681.
This API only works in very limited circumstances -- relying on
it will invariably lead to leaks. The correct way to "free" something
from the pool is to use the checkpoint API.
When row data split across multiple packets, allocate a temporary
buffer that can be reallocated, and only copy into the row buffer
pool arena once we know the final size. This avoids quadratic
memory usage for very large results.
FilesystemIterator::FOLLOW_SYMLINKS is currently treated as a directory
key mode flag, even though it does not change the way that the key
during iteration is set. To address this, FOLLOW_SYMLINKS has been
converted into an OTHER flag.
Closes GH-6695.
* allow the user to specify a folder where files that can be sent
via LOAD DATA LOCAL can exist
* add mysqli.local_infile_directory for mysqli
(ignored if mysqli.allow_local_infile is enabled)
* add PDO::MYSQL_ATTR_LOCAL_INFILE_DIRECTORY for pdo_mysql
(ignored if PDO::MYSQL_ATTR_LOCAL_INFILE is enabled)
* add related tests
* fixes for building with libmysql 8.x
* small improvement in existing tests
* update php.ini-[development|production] files
Closes GH-6448.
Co-authored-by: Nikita Popov <nikic@php.net>
Adjust zend_resolve_class_name() to not resolve special class names.
This avoids the need to only call this function after a preliminary
check for non-default fetch types. Doing so is somewhat fragile
when dynamic class names are involved.
Fixes oss-fuzz #31139.
A secret can be passed through the options array. The length is
currently in the range of 136 to 256 bytes. The concerned algos are
already marked as non serializable.
Signed-off-by: Anatol Belski <ab@php.net>
For fake closures, we need to share static variables with the
original function, not work on a separate copy. Calling a function
through Closure::fromCallable() should have the same behavior as
calling it directly.
When a method is inherited, the static variables will now always
use the initial values, rather than the values at the time of
inheritance. As such, behavior no longer depends on whether
inheritance happens before or after a method has been called.
This is implemented by always keeping static_variables as the
original values, and static_variables_ptr as the modified copy.
Closes GH-6705.
The UNDEF marker here is important to prevent the creation of
a reference to the property currently being overwritten, which
would then leak.
This fixes oss-fuzz 6029559193534464, which was incorrectly
merged into oss-fuzz #30584 (which is reported at
https://github.com/google/oss-fuzz/issues/5211).
This restores the previous behavior for this case. We'll continue
to use the mangled name, even if it does not correspond to a
declared property.
This also fixes an assertion failure for the case of property
overwrite, as the add_new was not guaranteed to be "new" previously.
Fixes oss-fuzz #31045.
Now that we track header changes, a change to zend_vm_opcodes.h
requires rebuilding essentially the whole tree. In most cases,
changes to zend_vm_def.h don't actually result in any header
changes though.
Avoid touching the file by first checking whether the to be written
contents are actually the same.
- fixed array style for $vm_kind_name and $op_types
- deleted unnecessary else statement for is_hot_helper function
- fixed concatenation style
- fixed style of using whitespace between "(" and switch/foreach - replaced !is_null with "!=="
Closes GH-6704.
Add CURLStringFile class which works similarly to CURLFile, but
uploads a file from a string rather than a file. This avoids the
need to create a temporary file, or use of a data:// stream.
Basic usage:
$file = new CURLStringFile($data, 'filename.txt', 'text/plain');
curl_setopt($curl, CURLOPT_POSTFIELDS, ['file' => $file]);
Closes GH-6456.
Otherwise we may have inconsistent refcounts after OOM.
I expect this problem is much more prevalent, but this at least
fixes some string/array separation cases.
Fixes oss-fuzz #30999.
The C compiler sees that a dynamic function is being called, so it cannot infer
that iter->funcs has not changed.
This results in more assembly instructions and slightly more time to execute that code
path.
Unpacking traversables to arrays(`ZEND_ADD_ARRAY_UNPACK`),
starting foreach loops (`ZEND_FE_FETCH*`), etc. are affected.
```
<?php
/*
* Before: 1.576 seconds
* After: 1.474 seconds
*/
function example() {
$start = hrtime(true);
$it = new SplFixedArray(1000);
$total = 0;
for ($i = 0; $i < 100000; $i++) {
$total += count([...$it]);
}
$end = hrtime(true);
printf("Elapsed: %.6f\n", ($end - $start) / 1_000_000_000);
}
example();
```
This deprecates passing null to non-nullable scale arguments of
internal functions, with the eventual goal of making the behavior
consistent with userland functions, where null is never accepted
for non-nullable arguments.
This change is expected to cause quite a lot of fallout. In most
cases, calling code should be adjusted to avoid passing null. In
some cases, PHP should be adjusted to make some function arguments
nullable. I have already fixed a number of functions before landing
this, but feel free to file a bug if you encounter a function that
doesn't accept null, but probably should. (The rule of thumb for
this to be applicable is that the function must have special behavior
for 0 or "", which is distinct from the natural behavior of the
parameter.)
RFC: https://wiki.php.net/rfc/deprecate_null_to_scalar_internal_arg
Closes GH-6475.
This changes ini_set() to accept all scalar types
(string|int|float|bool|null) for the new value. The idea here is
that while the INI system ultimately works with strings, its value
interpretation is designed to be consistent with PHP's casting rules,
e.g. "1" and "" are interpreted as boolean true and false respectively.
I personally believe that writing ini_set('precision', 10) makes more
sense than ini_set('precision', '10'), and find strict_types to be
unnecessarily pedantic here.
Closes GH-6680.
Some upcoming changes like https://wiki.php.net/rfc/deprecate_null_to_scalar_internal_arg
will make it somewhat inconvenient to determine whether a given
function invocation will generate a diagnostic. Rather than trying
to exclude this in advance, call the function with diagnostics
suppressed, and check whether anything was thrown.
This adds a new EG flag that is kept specific to the SCCP use-case.
This does not use the error_cb hook as it is a (non-TLS) global,
and doesn't fully suppress error handling besides.
Test this by removing the in advance checks for implode and array_flip.
Avoid the overhead of a call and checking types
when the argument is definitely an array.
Avoid the overhead of gc when `__destruct` won't get called.
This seemed cheap enough to check for in the jit.
Because of https://wiki.php.net/rfc/restrict_globals_usage
we can be sure in the ZEND_COUNT handler that the array count does not have to
be recomputed in php 8.1.
The below example took 0.854 seconds before the optimization,
and 0.564 seconds after the optimization, giving the same result
```php
<?php
/** @jit */
function bench_count(int $n): int {
$total = 0;
$arr = [];
for ($i = 0; $i < $n; $i++) {
$arr[] = $i;
$total += count($arr);
}
return $total;
}
function main() {
$n = 1000;
$iterations = 50000;
$start = microtime(true);
$result = 0;
for ($i = 0; $i < $iterations; $i++) {
$result += bench_count($n);
}
$elapsed = microtime(true) - $start;
printf("Total for n=%d, iterations=%d = %d, elapsed=%.3f\n", $n, $iterations, $result, $elapsed);
}
main();
```
Before
```asm
mov $0x7feb8cf8a858, %r15
mov $ZEND_COUNT_SPEC_CV_UNUSED_HANDLER, %rax
call *%rax
```
After
```asm
mov 0x70(%r14), %rdi - Copy the count from the `zend_array*` pointer
mov %rdi, (%rax) - Store the count in the destination's value
mov $0x4, 0x8(%rax) - Store IS_LONG(4) in the destination's type
```
And add tracing jit support
Closes GH-5584
This is a new transparent technology that eliminates overhead of PHP class inheritance.
PHP classes are compiled and cached (by opcahce) separately, however their "linking" was done at run-time - on each request. The process of "linking" may involve a number of compatibility checks and borrowing methods/properties/constants form parent and traits. This takes significant time, but the result is the same on each request.
Inheritance Cache performs "linking" for unique set of all the depending classes (parent, interfaces, traits, property types, method types involved into compatibility checks) once and stores result in opcache shared memory. As a part of the this patch, I removed limitations for immutable classes (unresolved constants, typed properties and covariant type checks). So now all classes stored in opcache are "immutable". They may be lazily loaded into process memory, if necessary, but this usually occurs just once (on first linking).
The patch shows 8% improvement on Symphony "Hello World" app.
This only moves the files, adjusts the build system, exports APIs
and does minor fixups to make sure the code builds.
This does not yet try to make the optimizer usable independently
of opcache.
Closes GH-6642.
To avoid increasing the size of parser stack elements by storing
size_t offset and length, this instead only stores the start
offset (or rather pointer now) and determines the length of the
identifier in zend_lex_tstring.
Print "Cannot use empty array entries in keyed array assignment"
instead of "Cannot mix keyed and unkeyed array entries in assignments"
for a leading comma.
Rename MYSQLI_REFRESH_SLAVE to MYSQLI_REFRESH_REPLICA in line with
upstream change in MySQL. The old name is retained for
backwards-compatibility reasons, and may be deprecated/removed in
the future.
Closes GH-6632.
This function tests if an array contains only sequential integer keys. While
list isn't an official type, this usage is consistent with the community usage
of "list" as an annotation type, cf.
https://psalm.dev/docs/annotating_code/type_syntax/array_types/#lists
Rebased and modified version of #4886
- Use .stub.php files
- Add opcache constant evaluation when argument is a constant
- Change from is_list(mixed $value) to array_is_list(array $array)
RFC: https://wiki.php.net/rfc/is_list
Co-Authored-By: Tyson Andre <tysonandre775@hotmail.com>
Co-Authored-By: Dusk <dusk@woofle.net>
Closes GH-6070
Instead of manually maintaining the data in eaw_table.h, it is now automatically
generated by ucgendat/ucgendat.php, using the EastAsianWidth.txt file from
the Unicode Consortium.
Something must be said about the deleted test case. Back in 2004, someone
noticed that `mb_strwidth` didn't comply with Unicode 4.0. A test case was
added to expose the problem. Well, time keeps moving on, and with the changing
years, new Unicodes are born and old Unicodes die. Some characters which were
counted as double-width in Unicode 4.0 are no longer such in Unicode 13.0,
which renders the test case obsolete.
At the same time, make a couple of spelling/grammar fixes in ucgendat.php.
The extension name should match the name of the ext/ directory,
otherwise it will not get picked up by run-tests. It would be possible
to remap this in run-tests, but I think it's better to rename the
extension to follow the standard format. Other extensions also
use underscore instead of hyphen (e.g. pdo_mysql and not pdo-mysql).
Of course, the ./configure option remains hyphenated.
Closes GH-6613.
This makes --verify also check @implementation-alias. Failures are
ignored using @no-verify instead. Some mistakes have been made that
would have been caught by this...
Closes GH-6615.
Some functions have multiple aliases, while the $aliasMap can
only record one.
Methodsynopsis generation probably shouldn't use it either, but
at least that case seems to only be interested in whether there
is an alias at all.
* PHP-8.0:
Sync datefmt_get_calendar_object signature
Sync intlcal_create_instance() and IntlCalendar::createInstance()
Sync date_diff and DateTime::diff return type
From libcurl version 7.62.0 and later, it supports DNS-over-HTTPS with
[`CURLOPT_DOH_URL`](https://curl.se/libcurl/c/CURLOPT_DOH_URL.html) option.
This adds integration with the `CURLOPT_DOH_URL` option if libcurl version
is >= 7.62.0 (0x073E00).
For reference, Ubuntu 20.04+ `libcurl4-openssl-dev`-based PHP builds use Curl 7.68.
Closes GH-6612.
This flag indicated that an encoding was 'multi-byte'; it can use a variable
number of bytes to encode each character. As it turns out, we don't actually
need to check this flag anywhere, so it's better to remove it.
MicroSoft invented three encodings very similar to ISO-2022-JP/JIS7/JIS8, called
CP50220, CP50221, and CP50222. All three are supported by mbstring.
Since these encodings are very similar, some code can be shared. Actually,
conversion of CP50220/1/2 to Unicode is exactly the same operation; it's when
converting from Unicode to CP50220/1/2 that some small differences arise in how
certain katakana are handled.
The most important common code was a function called `mbfl_filt_wchar_jis_ms`.
The `jis_ms` part doubtless refers to the fact that these encodings are modified
versions of 'JIS' invented by 'MS'. mbstring also went a step further and exported
'JIS-ms' to userland as a separate encoding from CP50220/1/2. If users requested
'JIS-ms' conversion, they got something like CP50220/1/2, minus their special
ways of handling half-width katakana when converting from Unicode.
But... that 'encoding' is not something which actually exists in the world outside
of mbstring. CP50220/1/2 do exist in MicroSoft software, but not 'JIS-ms'.
For a text encoding conversion library, inventing new variant encodings and
implementing them is not very productive. Our interest is in handling text
encodings which real people actually use for... you know, storing actual text
and things like that.
CP50220 is a variant of ISO-2022-JP invented by MicroSoft, which handles some
Unicode characters which are not representable in ISO-2022-JP by converting
them to similar characters which are representable.
What, then, is CP50220-raw? An Internet search turns up absolutely nothing.
Reference works which I consulted don't say anything about it. Other text
conversion libraries don't support it.
From looking at the code: It's just the same as CP50220, but it accepts
unmapped JIS X 0208 characters passed through from other Japanese encodings
and silently encodes them using the usual ISO-2022-JP escape sequence and
representation for JIS X 0208 characters.
It's hard to see how this could be useful. OK, let me come out and say it:
it's _not_ useful. We can confidently jettison this (mis)feature.
This is slightly more aggressive about rejecting obviously incorrect
element counts. Previously the number of elements was allowed to
match the number of characters. Now it is the number of characters
divided by two (this can actually be increased further to at least 4).
This doesn't really matter in the grand scheme of things (as it
just cuts maximum memory usage by half), but should fix
oss-fuzz #29356.
We're starting to see a mix between uses of zend_bool and bool.
Replace all usages with the standard bool type everywhere.
Of course, zend_bool is retained as an alias.
debug_zval_dump() currently prints refcount 1 for interned strings
and arrays, which does not really reflect the truth. These values
are not refcounted, so the refcount is misleading. Instead print
an "interned" tag.
Closes GH-6598.
The extensions are build as shared to only check that they compile,
without running tests. The OCI8 extension does not properly SKIPIF
no database is available.
It should be noted that if we do want to also test these, then
(apart from running a database) it will also be necessary to configure
with LIBS="-Wl,--disable-new-dtags" in order to force the use of RPATH
instead of RUNPATH, the latter of which does not affect dlopened
libraries. Using LD_LIBRARY_PATH does not mesh well with our test
suite.
Closes GH-6604.
This function is unused in php-src, and has somewhat dubious
semantics, especially since we switched convert_to_long to not
use strtol for the base 10 case.
If you want to convert strings from a different base, use
ZEND_STRTOL directly.
Unicode has a range of 'private' codepoints which individual applications can
use for their own purposes. When they were inventing CP932, MicroSoft mapped
these 'private' or 'user' codepoints to ten new rows added to the JIS X 0208
character table. (JIS X 0208 is based on a 94x94 table; MS used rows 95-114
for private characters.)
`mbfl_filt_conv_wchar_jis_ms` converted these private codepoints to rows 85-94
rather than 95-114. The code included a link to a document on the OpenGroup
web site, dating back to 1996 [1], which proposed mapping private codepoints to
these rows. However, that is not consistent with what mbstring does when
converting CP5022x to Unicode.
There seems to be a dearth of information on CP5022x on the web. However, I
did find one (Japanese-language) page on CP50221, which states that it maps
kuten codes 0x7F21-0x927E to the 'private' Unicode codepoints [2].
As a side note, using rows higher than 95 does seem to defeat one purpose of
using an ISO-2022-JP variant: ISO-2022-JP was specifically designed to be
"7-bit clean", but once you go beyond row 95, the ku codes are 0x80 and up,
so 8 bits are needed.
[1] https://web.archive.org/web/20000229180004/http://www.opengroup.or.jp/jvc/cde/ucs-conv.html
[2] https://www.wdic.org/w/WDIC/Microsoft%20Windows%20Codepage%20%3A%2050221
Essentially, CP5022{0,1,2} are to CP932 as ISO-2022-JP is to Shift-JIS.
As Shift-JIS and ISO-2022-JP both encode characters from the JIS X 0208 charset,
CP932 and CP5022x both encode characters from JIS X 0208 _plus_ extra characters
added as MicroSoft vendor extensions.
Among the added characters are a number of symbols which MS put in the 13th row
of the 94x94 character table. (In JIS X 0208, that row is empty.)
mbfilter_cp50220x.c had an `if` clause which was intended to handle the
conversion of characters in that 13th row, but it was dead code, as the previous
clause was always true in those cases. The solution is to reverse the order of
those two clauses (just as they already appeared in mbfilter_cp932.c).
Don't allow escape sequences to start in the middle of a multibyte character.
Also, don't silently pass through illegal bytes which appear where the 2nd
byte of a multibyte character should be.
Previously, in ISO-2022-JP/JIS7/JIS8, if an escape sequence (starting with 0x1B)
appeared where the 2nd byte of a multibyte character should have been, mbstring
would forget all about the truncated multibyte character and happily accept the
escape sequence. However, such sequences are not legal and should be flagged as
errors.
Also, any other illegal bytes appearing where the 2nd byte of a multibyte
character was expected were just passed through quietly to the output. Fix that.
Also add a test suite for both ISO-2022-JP and JIS7/JIS8. (These are extremely
similar encodings; JIS7 and JIS8 are variants of ISO-2022-JP. mbstring's 'JIS'
is actually a combination of JIS7 _and_ JIS8, since the extensions which each
one adds to ISO-2022-JP are disjoint.)
In other legacy Japanese encodings like Shift-JIS, we are now using a specific
JISX 0208 character for the Unicode overline (U+203E). Previously, the single
byte 0x7E was used, but an ASCII 0x7E does not represent an overline, so this
was changed.
However, JIS7/JIS8 can represent characters in the JISX 0201 character set as
well. That character set also includes an overline character, which takes less
bytes to encode than the corresponding JISX 0208 character, so we'll use it.
This is what mbstring had been doing for a long time; but it changed as a
side effect of the recent changes to how U+203E is encoded in Shift-JIS, etc.
So change it back.
These encodings have multiple modes which can be selected via escape sequences.
The default starting mode is ASCII. If a string _ends_ in a different mode, we
emit a 'redundant' escape sequence to switch back to ASCII.
If the resulting string is never concatenated with other strings, that extra
escape sequence serves no purpose. But if the resulting string is concatenated
with other strings of the same encoding, it ensures that the resulting string
will be valid.
This issue dates back to some commits I merged recently, which made encodings
like Shift-JIS-2004 use appropriate JIS X 0208 characters to represent
backslashes and tildes, rather than single-byte characters which are used in
those encodings with a different meaning (for example, in these encodings,
0x5C is used for a halfwidth Yen sign, rather than a backslash).
There was an unintended side effect: ISO-2022-JP-2004 was also made to
represent backslashes and tildes using JIS X 0208 characters. However,
ISO-2022-JP explicitly includes ASCII as one of its selectable character sets,
and ISO-2022-JP-2004 is just an extension of ISO-2022-JP. So when converting
text to ISO-2022-JP-2004, we can convert Unicode backslashes and tildes to ASCII
rather than using the corresponding JIS X 0208 characters.
This function is unused, and also not particularly useful now that
PHP no longer prints doubles in a locale-sensitive way unless
someone really goes out of their way to force it.
Historically, the _ex variants separated the zval first, if a
conversion was necessary. This distinction no longer makes sense
since PHP 7.
The only difference that was still left is that _ex checked whether
the type is the same first, but the usage of these macros did not
actually distinguish on whether such an inlined check is valuable
or not in a given context.
Also drop the unused convert_to_explicit_type macros.
Separation can only possibly make sense for array parameters
(or something that can contain arrays, like zval parameters). It
never makes sense to separate a bool.
The deref parameters are also of dubious utility, but leaving them
for now.
The name doesn't correspond to what it does at all, and all the
existing usages appear to be unnecessary.
Usage of this macro can be replaced by ZVAL_DEREF + Z_TRY_ADDREF_P.
- ZCG(bind_hash) is not used anymore
- zend_accel_function_hash_copy() and zend_accel_function_hash_copy_from_shm() are the same
- zend_accel_class_hash_copy() and zend_accel_class_hash_copy_from_shm() are almost the same
The implementation bundles the xxHash v0.8.0 release and includes all the variants
- xxh32, 32-bit wide
- xxh64, 64-bit wide
- xxh3, 64-bit wide
- xxh128, 128-bit wide
An initial hash state can be passed through the options arrray. An additional
functionality not targeted in this implementation is the secret support in xxh3
and xxh128. That can be added at a later point.
The serialization for xxh3 and xxh128 should not be implemented, as the
state would contain the secret. Despite the xxHash is a non crypto
algorithm, the secret would be serialized as plain text which would be
insecure.
Closes GH-6524
Signed-off-by: Anatol Belski <ab@php.net>
Previously, two useless FE_RESET_R and FE_FREE would be left over whether the empty array
was from a literal, a variable, or a class constant.
This doesn't pick up the RESET_RW case due to a weakness in our "may throw"
modeling. (for foreach by reference).
Co-Authored-By: Nikita Popov <nikita.ppv@gmail.com>
using https://gist.github.com/nikic/58d367ad605e10299f5433d2d83a0b5b
Closes GH-4949
This restricts allowed usage of $GLOBALS, with the effect that
plain PHP arrays can no longer contain INDIRECT elements.
RFC: https://wiki.php.net/rfc/restrict_globals_usage
Closes GH-6487.
Introduced by the recent switch to a zend_object. Unserialize the
object into a tmp_var to avoid leaving behind a stack reference.
Fixes oss-fuzz #29271.
This is simpler than going through curl, because we can manage
the memory more easily (in particular, this removes the need for
to_free->str). Additionally this allows the CURLOPT_PRIVATE data
to have any type, instead of being implicitly cast to string.
For these callbacks a non-null callback already indicates that
a user callback should be used. We don't need an additional
member to tell us the same thing.
Track an 8-byte pointer as the key instead of 16-byte zval.
(4 byte on 32-bit builds)
The memory used for a set of 1 million objects decreased from
114 MiB to 106 MiB.
Use the new `Z_PARAM_*` APIs to parse the `zend_object*` pointer directly
Closes GH-6566
This was using strcmp instead of zend_string_equals_literal.
As a result, the property count didn't match the number of properties
being serialized if properties started with
"__PHP_Incomplete_Class\0" (unlikely)
(before, `'O:8:"Missing_":1:{}'` would be serialized, which failed to
unserialize)
Everywhere else expects the MAGIC_MEMBER to match exactly,
and this should use zend_string_equals_literal as an example for other code.
This has used strcmp since 2004 in deb84befae
Closes GH-6555
This decreases the memory usage of SplFixedArrays by 32 bytes per object
on 64-bit systems (use 1 null pointer instead of 5 null pointers)
If allocating a lot of arrays of size 1, memory usage was 19.44MiB before this
change, and 16.24MiB after the change.
Existing tests continue to pass.
Subclassing SplFixedArray is already inefficient and rarely done.
It checks for the existence of 5 methods every time a subclass is instantiated.
(and has to switch back from C to the php vm to call those methods)
Closes GH-6552
This exports a php_getenv() API which will fetch an environment
variable in a thread-safe manner (assuming all other environment
manipulations are thread-safe ... ha ha ha).
Closes GH-6571.
Use php_random_bytes() to generate the MT19937 seed if none is
explicitly given. Also use it to generate more_entry for uniqid().
These changes should not impact user-observable behavior apart
from result statistics.
Closes GH-6520.
* PHP-8.0:
Fix test name
[ci skip] Update NEWS
[ci skip] Update NEWS
[ci skip] Fix order
[ci skip] Unpdate NEWS
Fix bug #79405 - gethostbyname() silently truncates after a null byte
Fix#77423: parse_url() will deliver a wrong host to user
Fix#77423: parse_url() will deliver a wrong host to user
When PDO firebird is loaded and these posix tests are run with
parallelism, LeakSanitizer will occasionally segfault. I was not
able to root cause this issue, as I failed to reproduce it in
simpler circumstances. I don't think it's an issue on our side
though.
To avoid that `parse_url()` returns an erroneous host, which would be
valid for `FILTER_VALIDATE_URL`, we make sure that only userinfo which
is valid according to RFC 3986 is treated as such.
For consistency with the existing url parsing code, we use ctype
functions, although that is not necessarily correct.
In some cases, like spl_object_id, the code is simpler but equally efficient
after optimizations.
In other cases, like get_mangled_object_vars(), the compiler can't infer that
the object in the zval won't change.
Closes GH-6567
Disallows constructing an `IMAPConnection` class directly with `new IMAPConnection` construct, by throwing an `Error` exception if attempted.
`imap_open` is still the only way to create `IMAPConnection` objects.
Return integers and floats as native types if possible. As usual,
the old behavior can be restored by enabling ATTR_STRINGIFY_FETCHES.
Fixes bug #38334.
Similar to other resource to object migrations, `FTPConnection` class is not allowed to be constructed with `new FTPConnection`.
Related to b4503fbf88.
Closes GH-6533.
Instead of attempting to connect with an invalid password, connect as an anonymous user.
As this is disabled in our Dovecot configuration, this generates an error, and fails fast.
Instead of requiring the type to be determined in advance by the
describer function and then requiring get_col to return a buffer
of appropriate type, allow get_col to return an arbitrary zval.
See UPGRADING.INTERNALS for a more detailed description of the
change.
This makes the result fetching simpler, more efficient and more
flexible. The general possibility already existed via the special
PDO_PARAM_ZVAL type, but the usage was very inconvenient and/or
inefficient. Now it's possible to easily implement behavior like
"return int if it fits, otherwise string" and to avoid any kind
of complex management of temporary buffers.
This also fixes bug #40913 (our second highest voted bug of all
time, for some reason). PARAM_LOB result bindings will now
consistently return a stream resource, independently of the used
database driver.
I've tried my best to update all PDO drivers for this change, but
some of the changes may be broken, as I cannot test or even build
some of these drivers (in particular PDO dblib and PDO oci).
Fixes are appreciated -- a working CI setup would be even more
appreciated ;)
Firebird does not have "if exists", and the switch to exception
error mode broke the suppressed exec calls.
Firebird does have a "recreate table" command that effective
perform a "drop table if exists" beforehand.
We already install the necessary libs for the odbc extension, so
we may as well build PDO ODBC.
As we don't setup an ODBC server, the tests will be skipped.
Close GH-6528.
Besides our general desire to get rid of the legacy resource types,
this is particularly appealing for fileinfo, because there are already
respective objects.
Closes GH-5987.
This allows reusing an existing zend_string inside a memory stream
without reallocating. For non-readonly streams, the string will
only get separated on write.
This tests that mysqli and pdo_mysql build against libmysqlclient,
and that tests pass for pdo_mysql. mysqli has too many test failures.
This is not an officially supported configuration.
There doesn't seem to be any compelling reason to implement this
in mysqlnd rather than mysqli. It's just a loop over fetch_into.
This makes the function available under libmysqlclient as well,
and thus fixes bug #79372.
There were only two uses of non-zmm allocation functions left,
which really did not need to use the system allocator. Remove
them, and remove the system allocator based APIs.
Previously, PDO MySQL only fetched data as native int/float if
native prepared statements were used. This patch updates PDO to
have the same behavior for emulated prepared statements, and thus
removes the largest remaining discrepancy between these two modes.
Note that PDO already has a ATTR_STRINGIFY_FETCHES option to control
whether native types are desired or not. The previous output can
be restored by enabling this option.
Most of the tests make use of that option, because this allows the
tests to work under libmysqlclient as well, which currently always
returns string results (independently of whether native or emulated
PS are used).
This is a larger overhaul of the mysqlnd result set infrastructure:
* Drop support for two different types of buffered results sets
("c" and "zval"). Possibly these made sense at some earlier
time, but now (with minor adjustments) one option is strictly
worse than the other. Buffered result sets already buffer the
full row packets, from which zvals can be decoded. The "zval"
style additionally also buffered the decoded zvals. As result
sets, even buffered ones, are generally only traversed once,
this just ends up wasting memory. Now, a potentially useful
variation here would be to buffer the decoded zvals instead of
the row packets, but that's not what the code was doing.
* To make it really strictly better, pre-allocate the zval row
buffer and reuse it for all rows. Previously the "c" style always
allocated a new buffer for each row.
* The fetch_row API now provides a populated zval[]. The task of
populating an array is deferred to fetch_row_into, which also
avoids duplicating this code in multiple places. The fetch_row_c
API is also implemented on top of fetch_row now, rather than
duplicating large parts of the code.
* The row fetching code for prepared statements and normal result
sets has been mostly merged. These already used the same
infrastructure, but prepared statements used separate row
fetching functions that were nearly the same as the normal ones.
This requires passing the stmt into the result set, rather than
just a flag. The only part that remains separate is reading of
unbuffered results in the presence of PS cursors.
Retain the field, but always populate it with zero. This was
already the case for PS without length updating.
max_length has nothing lost in the field metadata -- it is a
property of the specific result set, and requires scanning the
whole result set to compute. PHP itself never uses max_length
with mysqlnd, it is only exposed in the raw mysqli API.
Keeping it for just that purpose is not worthwhile given the costs
involved. People who actually need this for some reason can easily
calculate it themselves, while making it obvious that the
calculation requires a full result set scan.
Merge it into free_result. There is a large number of different
free_* functions for result sets, let's avoid having one more.
Only difference is that it does not increment stats, and that
seems like a bug as free_stmt_result is still freeing a result.
Something odd was being done here, with the row packet having a
flag for whether it should allocate the zval buffer, which would
then get moved into the result set.
Keep the management of this buffer purely at the result set level.
This also allows us to easily reuse the same buffer for all results,
rather than allocating a new one for each fetch.
Use the same approach as for GMP: If no explicit directory is
given, assume it must be on the default include path and libdir.
Otherwise use the provided path. It does not look like libsybdb
has support for pkg-config.
`version_compare()` does a sloppy check for the `$operators` argument
which allows arbitrary abbreviations of the supported operators to be
accepted. This is both undocumented and unexpected, and could lead to
subtle BC breaks, if the order of the comparisions will be changed.
Therefore we change to strict comparisons.
Closes GH-6510.
The concrete need on this change is to support passing an initial seed
to the murmur hash. Passing a custom seed is important in terms of
randomizing the hash function.
The suggested implementation adds a HashTable parameter to all the
init callbacks. Further on, an array with custom arguments is accepted
from `hash` or `hash_init` from the user land. Currently several things
like `hash_hkdf` are not touched, as they don't need passing custom
args.
Some convenience macros have been added to the SHA/MD families of
functions, so the consuming code doesn't have to be changed widely.
Another way to implement this is to add another type of the init that
would accept a HT with arguments. However, that would still require
touching all the context structs in all the algos. That would also
increase the size of those structs. As an init function is called just
once, the way of modifying the existing init callback has been seen
as more preferrable.
Closes GH-6400.
Signed-off-by: Anatol Belski <ab@php.net>
Co-Developed-by: Nikita Popov <nikita.ppv@googlemail.com>
Signed-off-by: Nikita Popov <nikita.ppv@googlemail.com>
Acked-by: Michael Wallner <mike@php.net>
Reviewed-by: Máté Kocsis <kocsismate@woohoolabs.com>
Reviewed-by: Eddie Kohler <ekohler@gmail.com>
This would be either $end or "end of file" depending on bison
version. Explicitly specify "end of command" instead, which seems
more appropriate in context.
Fills out the array/object-property insert helpers for zend_array, zend_object, and zend_reference.
This adds the following matrix of 18 APIs
add_next_index_T()
add_index_T()
add_assoc_T()
add_assoc_T_ex()
add_property_T()
add_property_T_ex()
Where T in array, object, reference
Converted internal callsites currently doing an explicit object wrap.
* PHP-8.0:
Fix Bug #80438: imap_msgno() incorrectly warns and return false on valid UIDs in PHP 8.0.0
Rename XmlParser to XMLParser for consistency with XMLWriter/XMLReader
Except for vanilla Shift-JIS, where 0x7E is a halfwidth overline/macron.
As for Shift-JIS-2004, it has an added character (byte sequence 0x854A)
which was defined as a halfwidth macron in JIS X 0213:2000, so we use that.
By entering this character in the JIS X 0208 conversion table, we can
remove a bunch of explicit `if` clauses in different conversion filters.
It also means that U+FF5E can be converted into SJIS-mac now; I don't
know why this one SJIS variant rejected U+FF5E before, since 0x8160
means the same thing in SJIS-mac as the others.
Converting U+203E to 0x7E was especially wrong for CP932, where 0x7E
represents a tilde.
For vanilla Shift-JIS and Shift-JIS-2004, converting to 0x7E is acceptable,
since 0x7E does represent an overline/macron in those encodings.
Follow the same principle in CP51932, which is closely related to CP932.
When Microsoft created CP932 (their version of Shift-JIS), they explicitly
used bytes 0-0x7F to represent ASCII characters rather than JIS X 0201
characters.
So when converting Unicode to CP932, it is not correct to convert U+00A5
to CP932 0x5C. Fortunately, CP932 does have a multi-byte FULLWIDTH YEN SIGN
character which we can use instead.
CP51932 uses the same extended character set as CP932; while CP932 is
MicroSoft's extended version of Shift-JIS, CP51932 is their extended version
of EUC-JP. So the same reasoning applies to CP51932.
Shift-JIS-2004 is an extension of Shift-JIS, which uses 0x5C for the Yen
sign. Therefore, it is not correct to convert ASCII 0x5C (backslash) to
Shift-JIS-2004 0x5C (yen sign). JIS X 0208 does have a backslash, so we
can convert ASCII backslash to SJIS-2004 backslash instead.
From time immemorial, there has been confusion around the treatment
of 0x5C bytes on systems using legacy Japanese encodings. JIS X 0201
specified that 0x5C means a yen sign, and thus fonts on Japanese systems,
including early versions of Windows, displayed a 0x5C byte as a yen sign.
This meant that when ASCII text files were displayed on such systems,
what were meant to be backslashes would appear as yen signs. Japanese C
programmers could write character escapes using yen signs, and C compilers
built on the assumption that the input was ASCII would interpret these
escapes as desired. Likewise for shell scripts. Et cetera, et cetera...
Therefore, if the input to `mb_convert_encoding` is (for example) a C
program, and after converting to Shift-JIS-2004, the user wishes to feed
the output into a C compiler, *then* perhaps ASCII 0x5C should be mapped
to SJIS 0x5C. However, this scenario is ridiculous and will never happen.
A more realistic scenario might be: an article written in SJIS-2004 has
embedded Windows file paths (like 'C:\Program Files'), with yen signs used
as a path separator. If we convert SJIS-2004 0x5C to ASCII 0x5C, then the
path separators will be 'fixed' by the conversion.
For general written texts, it is much better to convert backslashes to...
backslashes. And yen signs, to yen signs.
Lots of problems here.
- Don't pass 'control' characters through silently in the middle of a
multi-byte character.
- Treat it as an error if a multi-byte character is truncated.
- For ESC sequences used to encode emoji on earlier Softbank phones, if an
invalid ESC sequence is found, don't pass it through. Rather, handle it as
an error and respect `mb_substitute_character`.
- In ranges used by mobile vendors for emoji, if a certain byte sequence
doesn't map to any emoji, don't emit a mangled value (actually a raw
(ku*94)+ten value, which may not even be a valid Unicode codepoint at all).
- When converting Unicode to SJIS-Mobile, don't mangle codepoints which fall
in the 2nd range of MicroSoft vendor extensions.
Some vendor-specific emoji have been mapped to standard Unicode codepoints
now, rather than 'private use area' codepoints. When the legacy code was
written, these codepoints may not have existed yet in the Unicode standard
which was current at that time.
Also do a major code cleanup -- remove dead code, rearrange what is left,
use some new macros and helper functions to make the code clearer...
These constants indicate that a text encoding uses 2+ bytes for each character,
and is either big endian or little endian (respectively). But nothing in
mbstring cares about the difference between MBFL_ENCTYPE_MWC2BE and
MBFL_ENCTYPE_MWC2LE.
(Actually, nothing cares about whether these flags are set at all...
maybe we should just remove them?)
These flags identify text encodings in mbstring which use a constant number of
bytes per character. While some parts of the code do use these flags, usually
to detect cases which can be optimized due to constant-width encoding, nothing
cares whether the encodings are 'LE' (little-endian) or 'BE' (big-endian).
So we can simplify things by combining constants.
Similarly to JIS X 0208, mbstring would pass kuten codes which are not mapped
in the JIS X 0212, JIS X 0213, or CP932 character sets through silently when
converting to another Japanese encoding.
Many Japanese encodings, such as JIS7/8, Shift JIS, ISO-2022-JP, EUC-JP, and
so on encode characters from the JIS X 0208 character set. JIS X 0208 is based
on the concept of a 94x94 table, with numbered rows and columns. However,
more than a thousand of the cells in that table are empty; JIS X 0208 does not
actually use all 94x94=8,836 possible kuten codes.
mbstring had a dubious feature whereby, if a Japanese string contained one of
these 'unmapped' kuten codes, and it was being converted to another Japanese
encoding which was also based on JIS X 0208, the non-existent character would
be silently passed through, and the unmapped kuten code would be re-encoded
using the normal encoding method of the target text encoding.
Again, this _only_ happened if converting the text with the funky kuten code
to a Japanese encoding. If one tried converting it to Unicode, mbstring would
treat that as an error.
If somebody, somewhere, made their own private extension to JIS X 0208, and
used the regular Japanese encodings like Shift JIS and EUC-JP to encode this
private character set, then this feature might conceivably be useful. But how
likely is that? If someone is using Shift JIS, EUC-JP, ISO-2022-JP, etc. to
encode a funky version of JIS X 0208 with extra characters added, then that
should be treated as a separate text encoding.
The code which flags such characters with MBFL_WCSPLANE_JIS0208 is retained
solely for error reporting in `mbfl_filt_conv_illegal_output`.
- Don't allow control characters to appear in the middle of a multi-byte
character. (This was a strange feature of mbstring; it doesn't make much
sense, and iconv doesn't allow it.)
- Treat truncated multi-byte characters as an error.
We already tried this in PHP 7.4, but ran into issues, because
alpine did not support pkg-config for libargon2 (or had a broken
pc file, not sure). The Alpine issue has been resolved in the
meantime, so let's give this another try.
`configure` for `phpize` builds on Windows creates Makefile and
config.pickle.h and includes the latter via the command line option
`/FI`. That implies that config.pickle.h is always included before
config.w32.h, which means that standard definitions always override
extension specific definitions, while it should be the other way round.
Therefore, we change the inclusion order by including config.pickle.h
at the end of config.w32.h if the former is available, and also make
sure to avoid any potential C4005 warnings by `#undef`ining the macros
before defining them.
Closes GH-6269.
- For consistency with UTF-16, UTF-32, and UCS-4, strip leading byte
order marks.
- Treat it as an error if string is truncated (i.e. has an odd number
of bytes).
To give the background on this issue, here is an excerpt from JAPANESE.txt,
from the Unicode Consortium:
Apple has defined a block of 32 corporate characters as "transcoding
hints." These are used in combination with standard Unicode characters
to force them to be treated in a special way for mapping to other
encodings; they have no other effect. Sixteen of these transcoding
hints are "grouping hints" - they indicate that the next 2-4 Unicode
characters should be treated as a single entity for transcoding. The
other sixteen transcoding hints are "variant tags" - they are like
combining characters, and can follow a standard Unicode (or a sequence
consisting of a base character and other combining characters) to
cause it to be treated in a special way for transcoding. These always
terminate a combining-character sequence.
The transcoding coding hints used in this mapping table are:
0xF860 group next 2 characters as a single entity for transcoding
0xF861 group next 3 characters as a single entity for transcoding
0xF862 group next 4 characters as a single entity for transcoding
0xF87A variant tag for "negative" (i.e. black & white reversed)
0xF87E variant tag for vertical form
0xF87F variant tag for other alternate form
For example, the Apple addition character 0x85AB is Roman numeral
thirteen. There is no single Unicode for this (although there are
standard Unicodes for Roman numerals 1-12). Using the grouping hint
0xF862 in combination with standard Unicodes, we can map this as
0xF862+0x0058+0x0049+0x0049+0x0049 (i.e. X + I + I + I).
Our SJIS-mac conversion code actually recognizes some special sequences
which start with an Apple 'transcoding hint'. However, if a transcoding
hint is misplaced and is not followed by one of the expected sequences,
we can just emit one error marker for the bad transcoding hint and then
process the following codepoint as normal.
When converting Unicode to MacJapanese, some special sequences of Unicode
codepoints are collapsed into a single SJIS character. When the implementation
sees a codepoint which *might* begin such a sequence, it is cached and examined
again after the next codepoint arrives.
If it turns out that it wasn't one of the 'special' sequences, then a 'fallback'
conversion table is consulted to convert the cached codepoint. Then we re-enter
the regular conversion code to convert the immediately following codepoint.
BUT, local variables need to be reinitialized properly when doing this!
Because the locals weren't reinitialized, the sad result was that some codepoints
would get chopped up into bit salad and emitted as something totally bogus
(which might not even be valid SJIS-mac text at all).
Since 1993, Unicode has had a specific codepoint for a fullwidth Yen sign.
Likewise, MacJapanese has separate kuten codes for halfwidth and fullwidth
Yen signs. But mbstring mapped _both_ Yen sign codepoints to the
MacJapanese fullwidth Yen sign.
It's probably more appropriate to map the 'ordinary' Yen sign to the
MacJapanese halfwidth Yen sign. Besides, this means that the conversion
between Unicode and MacJapanese is closer to being lossless and reversible.
Also, don't accept 1st bytes above 0xED, since none of the possible 2-byte
sequences starting with 0xEE and above are actually mapped to any character.
Unicode has 'combining' characters which join with another following character.
Japanese hiragana and katakana with the 'two dots' voice mark can be represented
in this way, with one Unicode character for the 'base' kana and another one which
adds the voice mark.
In SJIS-2004, however, there are dedicated characters for voiced and unvoiced
kana. So some special checks are done to identify sequences of Unicode characters
which need to be 'collapsed' into a single SJIS-2004 character.
If a kana is immediately followed by some other unrelated character, like a
Cyrillic letter, then the cached kana should be output 'as is' and we
proceed with encoding the unrelated character. When doing this, though,
we need to re-initialize local variables, or else the unrelated character
will be mangled in some cases.
If the 2nd byte of a 2-byte character is invalid, then mb_substitute_character()
should be respected. Instead, what mbstring was doing was 'swallowing' the
first byte, then emitting the 2nd byte as if it was an ASCII character.
Likewise, if the 2nd byte is missing, instead of just keeping quiet, report an
illegal character as specified by mb_substitute_character().
This faulty binary search would never reject values at the very high
end of the range being searched, even if they were not actually in
the table.
Among other things, this meant that some Unicode codepoints which do
not correspond to any character in JIS X 0213 would be converted to
bogus Shift-JIS-2004 values rather than being rejected.
Each flush function in a chain of mbstring conversion filters always
calls the next flush function in the chain. So it is not necessary to
explicitly flush the second filter in a chain. (Due to this bug, in many
cases, flush functions were actually being called three times.)
Previously, the unit tests for these text encodings covered all mappings
from legacy -> Unicode, and all _reversible_ mappings from Unicode -> legacy.
However, we should also test the few Unicode -> legacy mappings which
are not reversible.
mbstring had an 'identify filter' for almost every supported text encoding
which was used when auto-detecting the most likely encoding for a string.
It would run over the string and set a 'flag' if it saw anything which
did not appear likely to be the encoding in question.
One problem with this scheme was that encodings which merely appeared
less likely to be the correct one were completely rejected, even if there
was no better candidate. Another problem was that the 'identify filters'
had a huge amount of code duplication with the 'conversion filters'.
Eliminate the identify filters. Instead, when auto-detecting text
encoding, use conversion filters to see whether the input string is valid
in candidate encodings or not. At the same type, watch the type of
codepoints which the string decodes to and mark it as less likely if
non-printable characters (ESC, form feed, bell, etc.) or 'private use
area' codepoints are seen.
Interestingly, one old test case in which JIS text was misidentified
as UTF-8 (and this wrong behavior was enshrined in the test) was 'fixed'
and the JIS string is now auto-detected as JIS.
- Don't allow control characters to appear in the middle of a multi-byte
character. (A strange feature, or perhaps misfeature, of mbstring which is
not present in other libraries such as iconv.)
- When checking whether string is valid, reject kuten codes which do not
map to any character, whether converting from EUC-JP to another encoding,
or converting another encoding which uses JIS X 0208/0212 charsets to
EUC-JP.
- Truncated multi-byte characters are treated as an error.
- Reject otherwise valid kuten codes which don't map to anything in JIS X 0208.
- Handle truncated multi-byte characters as an error.
- Convert Shift-JIS 0x7E to Unicode 0x203E (overline) as recommended by the
Unicode Consortium, and as iconv does.
- Convert Shift-JIS 0x5C to Unicode 0xA5 (yen sign) as recommended by the
Unicode Consortium, and as iconv does.
(NOTE: This will affect PHP scripts which use an internal encoding of
Shift-JIS! PHP assigns a special meaning to 0x5C, the backslash. For example,
it is used for escapes in double-quoted strings. Mapping the Shift-JIS yen
sign to the Unicode yen sign means the yen sign will not be usable for
C escapes in double-quoted strings. Japanese PHP programmers who want to
write their source code in Shift-JIS for some strange reason will have to
use the JIS X 0208 backlash or 'REVERSE SOLIDUS' character for their C
escapes.)
- Convert Unicode 0x5C (backslash) to Shift-JIS 0x815F (reverse solidus).
- Immediately handle error if first Shift-JIS byte is over 0xEF, rather than
waiting to see the next byte. (Previously, the value used was 0xFC, which is
the limit for the 2nd byte and not the 1st byte of a multi-byte character.)
- Don't allow 'control characters' to appear in the middle of a multi-byte
character.
The test case for bug 47399 is now obsolete. That test assumed that a number
of Shift-JIS byte sequences which don't map to any character were 'valid'
(because the byte values were within the legal ranges).
There is no meaningful difference between these and UCS-{2,4}. They are
just a little bit more lax about passing errors silently. They also have
no known use.
Alias to UCS-{2,4} in case someone, somewhere is using them.
The script for the clean section was missing the $ Meta-character to only delete the default
mailbox when it's an exact match.
Reinstore parallel testing for more tests.
- Identify filter was completely wrong.
- Respect `mb_substitute_character` rather than converting invalid bytes to
Unicode 0xFFFD (generic replacement character).
- Don't convert Unicode 0xFFFD to a valid ARMSCII-8 character.
- When converting ARMSCII-8 to ARMSCII-8, don't pass invalid bytes through
silently.
Previously, `mb_check_encoding` did an awful lot of unneeded work. In order to
determine whether a string was valid or not, it would convert the whole string
into wchar (code points), which required dynamically allocating a (potentially
large) buffer. Then it would turn right around and convert that big 'ol buffer
of code points back to the original encoding again. Finally, it would check
whether any invalid bytes were detected during that long and onerous process.
The thing is, mbstring _already_ has machinery for detecting whether a string
is valid in a certain encoding or not, and it doesn't require copying any data
around or allocating buffers. Better yet, it can fail fast when an invalid byte
is found. Why not use it? It's sure a lot faster!
Further, the legacy code was also badly broken. Why? Because aside from
checking whether illegal characters were detected, it would also check whether
the conversion to and from wchars was lossless. But, some encodings have
more than one valid encoding for the same character. In such cases, it is
not possible to make the conversion to and from wchars lossless for every
valid character. So `mb_check_encoding` would actually reject good strings
in a lot of encodings!
One funny thing: while the original author used Unicode 0xFFFD (generic
replacement character) for invalid bytes in CP1251 and CP1252, for CP1254
they used 0xFFFE, which is not a valid Unicode codepoint at all, but is a
reversed byte-order mark. Probably this was by mistake.
Anyways,
- Fixed identify filter, which was completely wrong.
- Don't convert Unicode 0xFFFE to a random (but valid) CP1254 byte.
- When converting CP1254 to CP1254, don't pass invalid bytes through silently.
- Identify filter was as wrong as wrong can be.
- Invalid CP1251 byte 0x98 was converted to Unicode 0xFFFD (generic
replacement character), rather than respecting `mb_substitute_character`.
- Unicode 0xFFFD was converted to some random CP1251 byte.
- When converting CP1251 to CP1251, don't pass invalid bytes through silently.
The implementation is based on the upstream PMurHash. The following
variants are implemented
- murmur3a, 32-bit hash
- murmur3c, 128-bit hash for x86
- murmur3f, 128-bit hash for x64
The custom seed support is not targeted by this implementation. It will
need a major change to the API, so then custom arguments can be passed
through `hash_init`. For now, the starting hash is always zero.
Fixes bug #68109, closes#6059
Signed-off-by: Anatol Belski <ab@php.net>
Co-Developed-by: Michael Wallner <mike@php.net>
Signed-off-by: Michael Wallner <mike@php.net>
Also remove a bogus test (bug62545.phpt) which wrongly assumed that all invalid
characters in CP1251 and CP1252 should map to Unicode 0xFFFD (REPLACEMENT
CHARACTER).
mbstring has an interface to specify what invalid characters should be
replaced with; it's called `mb_substitute_character`. If a user wants to see
the Unicode 'replacement character', they can specify that using
`mb_substitute_character`. But if they specify something else, we should
follow that.
It's a bit surprising how much was broken here.
- Identify filter was utterly and completely wrong.
- Instead of handling invalid CP1252 bytes as specified by
`mb_substitute_character`, it would convert them to Unicode 0xFFFD
(generic replacement character).
- When converting ISO-8859-1 to CP1252, invalid ISO-8859-1 bytes would
be passed through silently.
- Unicode codepoints from 0x80-0x9F were converted to CP1252 bytes 0x80-0x9F,
which is wrong.
- Unicode codepoint 0xFFFD was converted to CP1252 0x9F, which is very wrong.
Also clean up some unneeded code, and make the conversion table consistent with
others by using zero as a 'invalid' marker, rather than 0xFFFD.
This testing mode executes the test multiple times in the same
process (but in different requests). It is primarily intended to
catch tracing JIT bugs, but also catches state leaks across
requests.
Closes GH-6365.
I don't like the previous behaviour where the bytes to hash change
every time the code changes, that may make it difficult to compare
hash() performance changes over time.
Use a fixed number instead, and allow passing an override for a
different length.
Closes GH-6386.
[ci skip]
Catch various errors such as the first part of a surrogate pair not being
followed by a proper second part, the first part of a surrogate pair appearing
at the end of a string, the second part of a surrogate pair appearing out
of place, and so on.
This broke one old test (Zend/tests/multibyte_encoding_003.phpt), which used
a PHP script encoded as UTF-16. The problem was that to terminate the test
script, we need the text: "\n--EXPECT--". Out of that text, the terminating
newline (0x0A byte) becomes part of the resulting test script; but a bare
0x0A byte with no 0x00 is not valid UTF-16.
Since we now treat truncated UTF-16 characters as erroneous, an extra '?' is
appended to the output as an 'illegal character' marker.
Really, if we are running PHP scripts which are treated as encoded in UTF-16
or some other arbitrary text encoding (not ASCII), and the script is not
actually a valid string in that encoding, inserting '?' characters into the
code which the PHP interpreter runs is a bad thing to do. In such cases, the
script shouldn't be treated as UTF-16 (or whatever) at all.
I wonder if mbstring's encoding detection is being used in 'non-strict' mode?
If Zip operations fails due to PHP error conditions before libzip even
has been called, there is no meaningful indication what failed; the
functions just return false, and the Zip status indicated that no error
did occur. Therefore we raise `E_WARNING` in these cases.
Closes GH-6356.
Port the main php_cli_server.inc to use ephemeral ports, thus
allowing CLI server tests to be parallelized.
A complication here is that we also need to give each test a
separate doc root, to avoid index.php files writing over each
other.
Closes GH-6375.
* PHP-8.0:
Checj type guards before loading values into CPU registers
Avoid register allocation for ASSIGN into aliased variable (it might be indirectly changed into IS_REFERENCE)
This macro is defined to zero as of PHP 5.0.0, and as the comment
indicates, is no longer relevant. Thus, we remove the definition and
all usages from the core and bundled extensions.
Closes GH-6351.
mbstring has a bad habit of passing invalid characters through silently
when converting to the same (or a "compatible") encoding.
For example, if you give it an invalid JIS X 0208 kuten code encoded with SJIS,
and try to convert that to EUC-JP, mbstring will just quietly re-encode the
invalid code in the EUC-JP representation.
At the same, some parts of the code (like `mb_check_encoding`) assume that
invalid characters will be treated as... well, invalid. Let's unbreak things
by actually catching errors and reporting them, instead of swallowing them.
Note that some text encoding conversion libraries, such as Solaris iconv
and FreeBSD iconv, map 0x30-0x39 to the Arabic script numerals rather than
the 'regular' Roman numerals. (That is, to Unicode codepoints 0x660-0x669.)
Further, Windows CP28596 adds more mappings to use the unused bytes in
ISO-8859-6.
There are some bytes in this encoding which are not mapped to any character.
Notably, MicroSoft added their own mappings for these 'unused' bits in their
version of Latin-3, called CP28593.
Interestingly, it looks like the original author intended to add an identify filter
for this encoding, but never did so. The needed struct is there, but was never added
to the list of identify filters in mbfl_ident.c.
There was one faulty test in the suite which only passed before because UTF-16 had no
identify filter. After this was fixed, it exposed the problem with the test.
- Make everything less gratuitously verbose
- Don't litter the code with lots of unneeded NULL checks (for things which
will never be NULL)
- Don't return success/failure code from functions which can never fail
- For encoding structs, don't use pointers to pointers to pointers for the
list of alias strings. Pointers to pointers (2 levels of indirection)
is what actually makes sense. This gets rid of some extraneous
dereference operations.
The check ensures that the decoded codepoint is between 0x10000-0x10FFFF,
which is the valid range which can be encoded in a UTF-16 surrogate pair.
However, just looking at the code, it's obvious that this will be true.
First of all, 0x10000 is added to the decoded codepoint on the previous
line, so how could it be less than 0x10000?
Further, even if the 20 data bits already decoded were 0xFFFFF (all ones),
when you add 0x10000, it comes to 0x10FFFF, which is the very top of the
valid range. So how could the decoded codepoint be more than 0x10FFFF?
It can't.
Add a get_gc method that can be implemented by drivers, which can
be used to add additional zvals to the GC buffer.
Implement GC support for PDO SQLite callbacks in particular.
Closes GH-6262.
See GCC bug 69602: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69602
which emits the warning for (errno == EWOULDBLOCK || errno == EAGAIN)
which is the correct way of handling errors as the value of EWOULDBLOCK
and EAGAIN is implementation defined.
Therefore introduce a new macro function PHP_IS_TRANSIENT_ERROR()
which handles the case when EWOULDBLOCK and EAGAIN are identical.
Thanks to @twose for the idea.
Loaded 100 of 5888 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Kamil Tekiela