Ordering of planets (mass and type)

Can I start out by jokingly complaining that you picked a rather complex system? We've found a lot of exoplanets, but there are not many that reside in complex systems like this. This is going to be a tough question. As Green predicted, Kepler data is useful here - Fang & Margo (2012) found that

75%"“80% of planetary systems have one or two planets with orbital periods less than 200 days

They also were able to plot data from a variety of parameters to come up with some graphs that could be used to make distribution curves. You can extrapolate from that, if you wish.

Anyway, I'm off track here. Mass distributions were covered in Mazeh et al. (1998) (which is almost certainly outdated, but a good analysis nonetheless) and Malhotra (2015). Using some orbital spacing parameters (which you can adjust, if you want), Malhotra found that the peak value of $\log m/M_{\oplus}$ occurs at about 0.6-1.0, with a standard deviation of 1.1-1.2. Not the greatest accuracy, but still pretty good.

Llambay et al. (2011) were able to come up with a mass-period distribution for exoplanets close to the star, which you can then use to come up with a decent distribution of masses at a given radius:

Most smaller planets have orbital periods longer than P~2.5 days, while higher masses are found down to P~1 day.

In short, more massive planets are closer in, while less massive planets are further out. Still, Llambay et al. only considered planets extremely close to their parent stars. For the rest of the system (i.e. planets farther out), I refer you to Jiang et al. (2007). I can't copy the mass- and period- histograms they gave (relating each one to the total number observed), nor can I copy the scatter plots, but they are incredibly helpful, especially as they considered a sample size of 233 exoplanets.

This graph, complied on Wikipedia from the Open Exoplanet Catalogue, is also helpful for an at-a-glance reference:

^{Image in the public domain.}

Something you must consider is planetary migration. I've written several answers on it across Stack Exchange (e.g. The Solar System Explosion in the Nice Model, Did Jupiter really make Earth (in)habitable, What gravitational impact would moving Jupiter to the inner solar system have on the outer?, etc. - the first focused on only one part, because Kyle Oman was already familiar with the it, hence the question), and others have written excellent answers elsewhere on Stack Exchange. By now, I'm sick and tired of typing the same thing up, so I refer you to the latter two posts I gave, as a starter. You need to include planetary migration because it will severely impact the orbits of the three gas giants in the system. Be careful that you have enough - my answer on Physics discusses just why a certain number is needed.

---

Planet mass to star ratio

No such ratio exists. You can have pretty much any (reasonable) combination you want. It all depends on the giant molecular cloud from which the star formed and the evolution of the protoplanetary disk. Anything can happen.

---

Number of planets

Fang & Margot are, once again, helpful. Weissbein et al. are also an excellent resource for this specific part. I once again wish I could figure out directly how to copy graphs and histogram without using imgur - I may use that later - but I can get around that. Unfortunately, they make three assumptions:

1. All planets in a system are exactly aligned
2. All of the stars and planets are identical
3. The Occupancy distribution of a planet existing at a given distance from its stellar host, f(r), is the same for all the stars which are capable of producing planets and is given by equation (1).

The third is not a problem, but the first two are (see my section on ecliptic plane confinement for a discussion on the first). Luckily, as I show later, that criterion can easily be met. The second one is the problem.

Anyway, Weissbein et al. find the probability, $P$, that a star hosts $m$ planets to be $$P(m)=\int_0^{\infty}\left[\frac{F(r)^m}{r^2m!}e^{-F(r)}\right]dr$$ where $r$ is radius and $$F(r)\equiv \int_0^r f(r')dr'$$ where $f(r')$ is a modified form of the general occupation probability function.

They then used this to create a table of the results, which I will not include at the moment, as I am not good with tables in Stack Exchange. However, predictably, the number of systems went down as the number of planets increased.

---

Ecliptic plane confinement

"Ecliptic plane confinement" can be discussed in terms of orbital inclination, generally denoted by $i$. In the case of most systems, this is close to zero degrees for most of the bodies involved (although Pluto has a high inclination).

The planets in the Solar System orbit in one plane, because everything formed out of a protoplanetary disk. The planets tend to stay that way because of a conservation of angular momentum. This can change in some cases - notably, Kepler-452b has a high angle of inclination (90 degrees!). As I wrote in my answer there, this may have happened for several reasons:

• The star's rotation axis was perturbed, just as Uranus's rotation axis was perturbed (although by different objects)
• The planet was perturbed by another object, either in the system (e.g. a planetoid) or a companion star. The Sun was formed with many other stars in a cluster; this happens for many stars.

The relevant papers on this subject are Crida & Batygin (2014) and Xue et al. (2014). There are other reasons for the change in orbital inclination of one planet, notably the Lidov-Kozai mechanism (see Lidov (1962) and Kozai (1962)). The Lidov-Kozai mechanism basically states that the eccentricity of an object's orbit can be changed by interactions with another (more massive) object, which also changes the orbital eccentricity of the first object. The angular momentum in the $z$-axis must be conserved here; it is the quantity $$L_z=\sqrt{1-e^2}\cos i$$ You can play around with this a bit to see what happens when different parameters are changed (you should be able to apply the orbital formulas given here). However, the model assumes that the perturber is much more massive than the perturbed object (Kozai's original analysis applied to perturbations of asteroids by Jupiter!). For larger bodies being perturbed, you would need a larger perturber. This makes it very difficult for planets. This could happen in a binary system where one star is more massive than another star, and the second star perturbs a planet moving around the larger star. It is, however, unlikely, and does not fit your model of one star.

It makes sense that either most or the orbits have high orbital inclinations - a result of a perturbation of the star's rotation axis or the protoplanetary disk - or low orbital inclinations. The Lidov-Kozai mechanism is not good for large systems. It is also important to note that it is periodic in nature. Once again quoting Fang & Margot,

In addition, over 85% of planets have orbital inclinations less than 3â—¦ (relative to a common reference plane).

They used a Rayleigh distribution to describe this: $$P(k)=\frac{k}{\sigma^2}e^{-k^2/\sigma^2}$$ where $\sigma$ is the parameter that determines the distribution of $k$. Notice the difference between a Rayleigh distribution and a normal distribution. An distribution for orbital eccentricity can be found in Kane et al. (2012).

---

Bringing it all together.

There's the raw information that we need. Here's the synthesis.

Is this the most likely arrangement, RxAxGx?

Well, it's unlikely for that many planets to form around a star, so technically, no. Three gas giants implies orbital migration, which could push them outwards, as in our Solar System, but be prepared to have a fourth be in there at the beginning, as some variants of the Nice Model require (the "5th gas giant").

Can a massive gas giant be orbiting near the star out of the ecliptic plane?

I stated earlier that the perturber generally needs to be more massive than the perturbed object, in classical models of the Kozai effect. This means that such an arrangement is unlikely to happen. A gas giant could be close to the star, sure, but not out of the ecliptic, if it was with a system of other planets that stayed in the ecliptic.

Can the habitable would be alone with some comets and asteroids?

Asteroids? Sure. Well, the habitable could not be in the asteroid belt, because then it would not have cleared its orbit and would be prone to collisions, which would quickly make the planet not so habitable!

The arrangement, on the whole, could happen.