You are essentially minimizing
subject to
. (The distance between the origin and any point
on the given surface is
, but
and
share the same critical points.)
Via Lagrange multipliers, we have Lagrangian
with partial derivatives (set equal to 0)
We assume
, which means
.
In the first case, we have
which means one of
must be positive, and the other is negative. From
we have
, so
So we get two critical points, (-5, 0, 5) and (5, 0, -5).
In the second case, if
, we get
which leads us to
i.e. we have two additional critical points (0, 5, 0) and (0, -5, 0).
At each of these points, we get respective distances from the origin of
, so the two closest points to the origin on the surface
are (0, 5, 0) and (0, -5, 0).