What unit of force is used in the universal gravitational constant?

That depends on the unit system used but in the SI (Systeme Internationale) system units are as shown here:
The gravitational constant ##G=6.67xx10^-11## ##m^3 kg^-1 s^-2##. Alternatively and equivalently the units can be written as ##Nm^2kg^-2## in which case the force unit is the newton (N).
The gravitational force expression is:
##F=(Gm_1m_2)/r^2##
If we rearrange to make ##G## the subject:
##G=(Fr^2)/(m_1m_2)##
We can replace the symbols with the appropriate units as follows:
##F## ##(N)##
##r## ##(m)##
##m## ##(kg)##
Then the units for ##G## we get are ##Nm^2kg^-2##.
One newton is one ##kgms^-2## which is how we translate between the two sets of units.