TBO increasing samples to 5, but staying with median is probably more effective. As median will throw out the high and the low and then average the middle 3. This should help reduce the effect of the first probe issue.
There are other ways to decrease the effect as well by adding dwell time between xy moves and first probe. I have discovered that if I use [smart_effector] instead of [probe] it allows adding two additional parameters: probe acceleration and recovery_time that seem to help as well…
probe_accel: 10
recovery_time: 0.4
# A delay between the travel moves and the probing moves in seconds.
You can also run two different probe accuracy tests that can gleen valuable information.
this one tests probe_accuracy in one spot, with or without temps (heat soaking)
e.g. you can see pi and controller temperatures on mine

and this one that tests multiple corners and center
probe_accuracy_test_suite.py
is a collection of tests to help checking probe accuracy, precision and drift under different conditions
edit: just sharing some useful info till the “ignore first probe” can be addressed.
Here is a simple test, sloooowd down. 
SET_SMART_EFFECTOR accel=1 recovery_time=0.4
probe_accuracy samples=10 probe_speed=1 sample_retract_dist=1 lift_speed=1
13:26:27
// SmartEffector:
// probing accelartion: 1.000
// probe recovery time: 0.4
13:26:27
// PROBE_ACCURACY at X:208.000 Y:205.000 Z:1.306 (samples=10 retract=1.000 speed=1.0 lift_speed=1.0)
13:26:29
// probe at 208.000,205.000 is z=0.307031
13:26:32
// probe at 208.000,205.000 is z=0.306250
13:26:35
// probe at 208.000,205.000 is z=0.306250
13:26:38
// probe at 208.000,205.000 is z=0.306250
13:26:40
// probe at 208.000,205.000 is z=0.306250
13:26:43
// probe at 208.000,205.000 is z=0.307031
13:26:46
// probe at 208.000,205.000 is z=0.307031
13:26:49
// probe at 208.000,205.000 is z=0.305859
13:26:52
// probe at 208.000,205.000 is z=0.307031
13:26:55
// probe at 208.000,205.000 is z=0.306250
13:26:55
// probe accuracy results: maximum 0.307031, minimum 0.305859, range 0.001172, average 0.306523, median 0.306250, standard deviation 0.000430