Introduction
I've included a diagram below.
You will note that, as CaptainBlack correctly said, for equilibrium the weight of the object must be directly over the point of contact.
In the diagram, you can see that
is the point that would be the centre of the flat face of the hemisphere,
is the axis of rotational symmetry of the original hemisphere,
is the point of contact with the inclined plane and
is the location of the centre of mass of the hemispherical object.
Geometry 
is a vertical line and since
is at right angles to the inclined plane we must have that the angle
is
. Also the angle that the flat face of the hemisphere makes with the horizontal is the same as the angle between
and the vertical i.e. we need to find the angle
. Let this angle,
, be
.
Using the sine rule for the triangle
, we find that
since
. Since we know
is acute we can rearrange this so
![\color[rgb]{0,0,1} \boxed{\theta = \arcsin \left[ \frac{r}{OC} \sin \alpha \right]}](http://www.mathhelpforum.com/math-help/latex2/img/5de4ed6cd634ec8e0b805b51303d4dee-1.gif "\color[rgb]{0,0,1} \boxed{\theta = \arcsin \left[ \frac{r}{OC} \sin \alpha \right]}")
.
So all we need to do is determine
.
Dimensions of the cube
The greatest cube that can be cut out of the hemisphere has its centre on
and so if we imagine expanding a small cube centred directly on top of
we will no longer be able to expand it when its height is such that its four other corners touch the curved surface of the hemisphere. This cube then has one face in the same plane as the flat surface of the hemisphere and the opposite face's corners touching the curved suface. At any one of these corners that meet the curved surface the distance from
must be
so using Pythagoras Theorem we must have
where
is the side length of the cube and
is the distance from
to the corresponding corner of the cube that lies in the plane of the flat surface of the hemisphere.
Again using Pythagoras we must have (for a cube) that
,
hence
,
![\color[rgb]{0,0,1} \boxed{\iff a = \sqrt{\frac{2}{3}} r}](http://www.mathhelpforum.com/math-help/latex2/img/a4a15ebfe70f1dbb14e5e9198e1ea355-1.gif "\color[rgb]{0,0,1} \boxed{\iff a = \sqrt{\frac{2}{3}} r}")
.
Centre of mass calculation
I'm going to assume that you know the result that the centre of mass of a uniform hemispherical solid is on the rotational axis of symmetry at
from
.
Let
be the density of the solid,
the mass and
be the distance of the centre of mass from
on
where the solid referred to is denoted by the subscripts
,
and
for hemisphere (uncut), cube and hemisphere with cube cut out respectively.
Then we must have (in terms of moments about
) that
and thus we can determine
since
,
,
,
,
.
So we have
,
with
.
Putting it together
Substituting our value of
into our trigonometric result gives us that
![\color[rgb]{1,0,1} \boxed{\theta = \arcsin \left[ \frac{8 \left(3 \pi - \sqrt{6} \right)}{9 \pi - 8} \sin \alpha \right]}](http://www.mathhelpforum.com/math-help/latex2/img/55d39e8f08cab688a7b3df67d962e510-1.gif "\color[rgb]{1,0,1} \boxed{\theta = \arcsin \left[ \frac{8 \left(3 \pi - \sqrt{6} \right)}{9 \pi - 8} \sin \alpha \right]}")
.
[SOLVED] Difficult statics question? ![\arcsin\left[\frac{8(3\pi - \sqrt{6})}{9\pi - 8}\sin \alpha\right]](http://www.mathhelpforum.com/math-help/latex2/img/85d34eb8185d9ec94784cabeff01777d-1.gif "\arcsin\left[\frac{8(3\pi - \sqrt{6})}{9\pi - 8}\sin \alpha\right]")
Which makes this confusing problem all the more confusing. 
Your problem solving abilities are uncanny. 
Linear Mode
