Look at the arc that each side cuts off.The idea. Take one side, say \(\displaystyle AB \). The line \(\displaystyle AB \) splits the plane in two, and the other two vertices \(\displaystyle C \) and \(\displaystyle D \) are on one of those sides — the inside of the quadrilateral is on that side too. The centre \(\displaystyle O \) lies on the same side of the chord \(\displaystyle AB \) as the
major arc. So:
\[O \text{ is inside } ABCD \iff \text{every side cuts off an arc smaller than a semicircle.} \]
Turning that into something you can measure. The arc \(\displaystyle AB \) that does not contain \(\displaystyle C \) and \(\displaystyle D \) is measured by the angle it subtends at \(\displaystyle C \) (or at \(\displaystyle D \)): \(\displaystyle \angle ACB = \tfrac{1}{2}\,\text{arc } AB \). Therefore, for each side, look at the angle it subtends at one of the two opposite vertices:
angle \(\displaystyle < 90^{\circ} \) (arc \(\displaystyle < 180^{\circ} \)): the centre is on the inside of that side;
angle \(\displaystyle = 90^{\circ} \) (arc \(\displaystyle = 180^{\circ} \)): that side is a diameter and the centre lies on it, at its midpoint;
angle \(\displaystyle > 90^{\circ} \) (arc \(\displaystyle > 180^{\circ} \)): the centre is on the far side of that side, so it is outside the quadrilateral.
So one honest method is: check all four such angles. All acute \(\displaystyle \Rightarrow \) centre inside.
The best way — you only ever need one measurement. The four arcs add up to \(\displaystyle 360^{\circ} \), so
at most one of them can exceed \(\displaystyle 180^{\circ} \). Moreover, the side cutting off that big arc is always the
longest side. Here is why. Suppose arc \(\displaystyle AB = a > 180^{\circ} \). Then the other three arcs satisfy \(\displaystyle b + c + d = 360^{\circ} - a < 180^{\circ} \). Now the chord \(\displaystyle AB \) also spans the
minor arc of measure \(\displaystyle 360^{\circ} - a = b+c+d \), and each of \(\displaystyle b, c, d \) is smaller than that. Among minor arcs (arcs under \(\displaystyle 180^{\circ} \)) of one circle, a longer arc has a longer chord. Hence \(\displaystyle AB \) is longer than \(\displaystyle BC \), \(\displaystyle CD \) and \(\displaystyle DA \).
So the quick test is:
1. Find the longest side of the quadrilateral.
2. Measure the angle it subtends at either of the two remaining vertices (the two answers are equal).
3. Acute \(\displaystyle \Rightarrow \) the centre is
inside; exactly \(\displaystyle 90^{\circ} \Rightarrow \) the centre is
on that side (its midpoint); obtuse \(\displaystyle \Rightarrow \) the centre is
outside, beyond that side.
Sanity checks. For a rectangle every side subtends an acute angle at the far vertices, and indeed the centre is inside (it is where the diagonals cross). For a thin quadrilateral whose four vertices are bunched together on a small piece of the circle, the long "closing" side subtends a fat obtuse angle, and the centre is far outside.
An equivalent way of saying it, if you prefer triangles: a diagonal cuts the quadrilateral into two triangles inscribed in the same circle, and the circumcentre of a triangle lies inside it exactly when the triangle is acute-angled. The centre lies inside the quadrilateral precisely when at least one of those two triangles is not obtuse.
Test the longest side: if it subtends an acute angle at the opposite vertices the centre is inside, if a right angle the centre lies on that side, and if an obtuse angle the centre is outside. Since at most one side can cut off a major arc, and that side must be the longest one, this single check settles it.