If \[\smallint \sqrt {ax + a\sqrt {ax - \mathop a\nolimits^2 } } dx,0 < a < 2 = \dfrac{2}{{3a}}\mathop {\{ ax + a\sqrt {ax - \mathop a\nolimits^2 } \} }\nolimits^{\dfrac{3}{2}} - \dfrac{{\sqrt a }}{2}\left| {A + B} \right| + c.\]Then
A) $A = [\{ (\sqrt {ax - \mathop a\nolimits^2 } + \dfrac{{\mathop a\nolimits^2 }}{2})\sqrt {ax + \mathop a\nolimits^2 \sqrt {ax - \mathop a\nolimits^2 } } \} ]$
B) $B = \log [\{ (\sqrt {ax - \mathop a\nolimits^2 } + \dfrac{{\mathop a\nolimits^2 }}{2}) + \sqrt {ax + a\sqrt {ax - \mathop a\nolimits^2 } } \} ]$
C) $A = \{ (\sqrt {ax - \mathop a\nolimits^2 } + \dfrac{{\mathop a\nolimits^2 }}{2})\sqrt {ax + a\sqrt {ax - \mathop a\nolimits^2 } } \} $
D) $B = \log \{ \sqrt {ax - \mathop a\nolimits^2 } + \dfrac{{\mathop a\nolimits^2 }}{2}\sqrt {ax + a\sqrt {ax - \mathop a\nolimits^2 } } \} $

Hint: When there is $\left( {\mathop x\nolimits^2 - \mathop a\nolimits^2 } \right)$ or $\left( {\mathop x\nolimits^2 + \mathop a\nolimits^2 } \right)$ in the integrant, then substitute $x = a\sec \theta $or $x = a\cos \theta $ accordingly.
Familiarization with trigonometric identities is a must to solve the calculus problems
To simplify the integration, try to use the substitution method.
A glance at the substitution method is given below.
*Integration by substitution
        The given integral $\int {f\left( x \right)} dx$can be transformed into another form by changing the independent variable x to t by substitution $x = g\left( t \right)$.
Consider $I = $ $\int {f\left( x \right)} dx$
Put $x = g\left( t \right)$ so that $\dfrac{{dx}}{{dt}} = g'\left( t \right)$
We write dx=g'tdt$dx = g'\left( t \right)dt$
Thus $I = $ $\int {f\left( x \right)} dx$$ = \int {f\left( {g\left( t \right)} \right)} g'\left( t \right)dt$
Convert the quadratic equation in perfect square to apply the integration of a particular function.
Key rule: Either substitute in the integration whose derivative is present or convert into a perfect square whose integral is already defined.

Complete step-by-step answer:
Step 1: given that:
I = \[\smallint \sqrt {ax + a\sqrt {ax - \mathop a\nolimits^2 } } dx\]= \[\dfrac{2}{{\mathop a\nolimits^{\dfrac{3}{2}} }}\mathop {\{ ax + a\sqrt {ax - \mathop a\nolimits^2 } \} }\nolimits^{\dfrac{3}{2}} - \dfrac{{\sqrt a }}{2}\left| {A + B} \right| + c\]
Provided: \[0 < a < 2\]
Step 2: To find
A and B
Step 3: solve the integration, I
I = \[\smallint \sqrt {ax + a\sqrt {ax - \mathop a\nolimits^2 } } dx\]
Let $x = a\mathop {\sec }\nolimits^2 \theta $
On differentiating both sides, we get
We know, $\dfrac{{d\left( {\sec x} \right)}}{{dx}} = \sec x\tan x$
$
  dx = 2a\sec \theta \cdot \sec \theta \tan \theta {\text{ }}d\theta \\
  {\text{ }} = 2a\mathop {\sec }\nolimits^2 \theta \tan \theta {\text{ }}d\theta \\
 $
On substituting, the integration becomes
$ \Rightarrow \smallint \sqrt {a \cdot a\mathop {\sec }\nolimits^2 \theta + a\sqrt {a \cdot a\mathop {\sec }\nolimits^2 - \mathop a\nolimits^2 } } 2a\mathop {\sec }\nolimits^2 \theta \tan \theta {\text{ }}d\theta $
$ \Rightarrow \smallint \sqrt {\mathop a\nolimits^2 \mathop {\sec }\nolimits^2 \theta + a\sqrt {\mathop a\nolimits^2 \mathop {(\sec }\nolimits^2 - 1)} } 2a\mathop {\sec }\nolimits^2 \theta \tan \theta {\text{ }}d\theta $
Using trigonometric identity: $\mathop {\tan }\nolimits^2 \theta + 1 = \mathop {\sec }\nolimits^2 \theta $
$\mathop {\because {\text{ }}\sec }\nolimits^2 \theta - 1 = \mathop {\tan }\nolimits^2 \theta $
$ \Rightarrow \smallint \sqrt {\mathop a\nolimits^2 \mathop {\sec }\nolimits^2 \theta + a\sqrt {\mathop a\nolimits^2 \mathop {(\tan }\nolimits^2 \theta )} } 2a\mathop {\sec }\nolimits^2 \theta \tan \theta {\text{ }}d\theta $
We know, \[\sqrt {{x^2}} = x\]
$ \Rightarrow \smallint \sqrt {\mathop a\nolimits^2 \mathop {\sec }\nolimits^2 \theta + \mathop a\nolimits^2 \tan \theta } 2a\mathop {\sec }\nolimits^2 \theta \tan \theta {\text{ }}d\theta $
Using trigonometric identity: $\mathop {\tan }\nolimits^2 \theta + 1 = \mathop {\sec }\nolimits^2 \theta $
$ \Rightarrow \smallint \sqrt {\mathop a\nolimits^2 (\mathop {\tan }\nolimits^2 \theta + 1) + \mathop a\nolimits^2 \tan \theta } 2a\mathop {\sec }\nolimits^2 \theta \tan \theta {\text{ }}d\theta $
Taking ${a^2}$ as common.
$ \Rightarrow \smallint \sqrt {\mathop a\nolimits^2 (\mathop {\tan }\nolimits^2 \theta + 1 + \tan \theta )} 2a\mathop {\sec }\nolimits^2 \theta \tan \theta {\text{ }}d\theta $
We know, $\sqrt {{a^2}} = a$, this a is multiplied with 2a.
$ \Rightarrow \smallint \mathop {2a}\nolimits^2 \sqrt {(\mathop {\tan }\nolimits^2 \theta + 1 + \tan \theta )} \mathop {\sec }\nolimits^2 \theta \tan \theta {\text{ }}d\theta $
Differentiation of $\tan \theta $ i.e. $\mathop {\sec }\nolimits^2 \theta $ is present outside the square root, hence simplify using substitution.
Take $\tan \theta = t$
On differentiating both sides, we get
$\mathop {\sec }\nolimits^2 \theta {\text{ }}d\theta = dt$
On substituting, the integration becomes
$ \Rightarrow \smallint \mathop {2a}\nolimits^2 \sqrt {(\mathop t\nolimits^2 + 1 + t)} {\text{ t }}dt$
Differentiation of $(\mathop t\nolimits^2 + 1 + t)$ is $2t + 1$, if $2t + 1$ was present in the integral then we can use the substitution method.
Combine 2 with t, Add +1, -1 to t will not affect the integration.
$ \Rightarrow \smallint \mathop a\nolimits^2 \sqrt {(\mathop t\nolimits^2 + 1 + t)} {\text{ (2t + 1 - 1) }}dt$
Separating the integration over $2t + 1$ and -1
$I = \smallint \mathop a\nolimits^2 \sqrt {(\mathop t\nolimits^2 + 1 + t)} {\text{ (2t + 1)}}dt - \smallint \mathop a\nolimits^2 \sqrt {(\mathop t\nolimits^2 + 1 + t)} {\text{ }}dt$
Let $\mathop I\nolimits_1 = \smallint \mathop a\nolimits^2 \sqrt {(\mathop t\nolimits^2 + 1 + t)} {\text{ (2t + 1)}}dt$
Differentiation of $(\mathop t\nolimits^2 + 1 + t)$ i.e. $2t + 1$is present, hence simplify using substitution.
Take $(\mathop t\nolimits^2 + 1 + t) = \tau $
On differentiating both sides, we get
$\left( {2t + 1} \right)dt = d\tau $
On substituting, the integration becomes
$\mathop I\nolimits_1 = \smallint \mathop a\nolimits^2 \sqrt \tau d\tau $
Using integration: \[\int {{x^n}} = \dfrac{{{x^{n + 1}}}}{{n + 1}}\]
\[
   \Rightarrow \mathop a\nolimits^2 \dfrac{{\mathop \tau \nolimits^{\dfrac{1}{2} + 1} }}{{\dfrac{1}{2} + 1}} + \mathop c\nolimits_1 \\
   \Rightarrow \mathop a\nolimits^2 \dfrac{{\mathop \tau \nolimits^{\dfrac{3}{2}} }}{{\dfrac{3}{2}}} + \mathop c\nolimits_1 \\
 \]
Here, $\mathop c\nolimits_1 $ is constant of integration
$ \Rightarrow \dfrac{2}{3}\mathop a\nolimits^2 \mathop \tau \nolimits^{\dfrac{3}{2}} + \mathop c\nolimits_1 $
Now, substituting back (t, x,) what we have let so far.
We know, $\tau = (\mathop t\nolimits^2 + 1 + t)$
$ \Rightarrow \dfrac{2}{3}\mathop a\nolimits^2 \mathop {(\mathop t\nolimits^2 + t + 1)}\nolimits^{\dfrac{3}{2}} + \mathop c\nolimits_1 $
We know, $t = \tan \theta $
${I_1} = \dfrac{2}{3}\mathop a\nolimits^2 \mathop {(\mathop {\tan }\nolimits^2 \theta + \tan \theta + 1)}\nolimits^{\dfrac{3}{2}} + \mathop c\nolimits_1 $
We know, $x = a\mathop {\sec }\nolimits^2 \theta $
\[
   \Rightarrow \dfrac{x}{a} = \mathop {\sec }\nolimits^2 \theta \\
   \Rightarrow \dfrac{x}{a} = \mathop {\tan }\nolimits^2 \theta + 1 \\
   \Rightarrow \mathop {\tan }\nolimits^2 \theta = \dfrac{x}{a} - 1 \\
   \Rightarrow \mathop {\tan }\nolimits^2 \theta = \dfrac{{x - a}}{a}{\text{ }}......{\text{ (1)}} \\
   \Rightarrow \tan \theta = \sqrt {\dfrac{{x - a}}{a}} \\
   \Rightarrow \tan \theta = \dfrac{{\sqrt {ax - \mathop a\nolimits^2 } }}{a}{\text{ }}......{\text{ (2)}} \\
 \]
On substituting the values of $\mathop {\tan }\nolimits^2 \theta $ and $\tan \theta $ in $\mathop I\nolimits_1 $, we get
$\mathop I\nolimits_1 = \dfrac{2}{3}\mathop a\nolimits^2 \mathop {\left( {\dfrac{{x - a}}{a} + \dfrac{{\sqrt {ax - \mathop a\nolimits^2 } }}{a} + 1} \right)}\nolimits^{\dfrac{3}{2}} + \mathop c\nolimits_1 $
Canceling a which is common in numerator and denominator
\[ \Rightarrow \dfrac{2}{3}\mathop a\nolimits^2 \mathop {\left( {\dfrac{{a(x - a)}}{{\mathop a\nolimits^2 }} + \dfrac{{a\sqrt {ax - \mathop a\nolimits^2 } }}{{\mathop a\nolimits^2 }} + 1} \right)}\nolimits^{\dfrac{3}{2}} + \mathop c\nolimits_1 \]
Taking LCM ${a^2}$ and simplifying.
\[
   \Rightarrow \dfrac{2}{3}\mathop a\nolimits^2 \mathop {\left( {\dfrac{{ax - \mathop a\nolimits^2 + a\sqrt {ax - \mathop a\nolimits^2 } + \mathop a\nolimits^2 }}{{\mathop a\nolimits^2 }}} \right)}\nolimits^{\dfrac{3}{2}} + \mathop c\nolimits_1 \\
   \Rightarrow \dfrac{2}{3}\mathop a\nolimits^2 \mathop {\dfrac{{\mathop {\left( {ax + a\sqrt {ax - \mathop a\nolimits^2 } } \right)}\nolimits^{\dfrac{3}{2}} }}{{\mathop a\nolimits^{2 \times \dfrac{3}{2}} }}}\nolimits^{} + \mathop c\nolimits_1 \\
   \Rightarrow \dfrac{2}{3}\mathop a\nolimits^2 \dfrac{{\mathop {\left( {ax + a\sqrt {ax - \mathop a\nolimits^2 } } \right)}\nolimits^{\dfrac{3}{2}} }}{{\mathop a\nolimits^3 }} + \mathop c\nolimits_1 \\
 \]
\[\mathop I\nolimits_1 = \dfrac{2}{{3a}}\mathop {\left( {ax + a\sqrt {ax - \mathop a\nolimits^2 } } \right)}\nolimits^{\dfrac{3}{2}} + \mathop c\nolimits_1 \]
Let $\mathop I\nolimits_2 = \smallint \mathop a\nolimits^2 \sqrt {(\mathop t\nolimits^2 + 1 + t)} {\text{ }}dt$
Convert the quadratic equation $(\mathop t\nolimits^2 + t + 1)$in the form of \[\mathop {(a + b)}\nolimits^2 \]
$
  (\mathop t\nolimits^2 + t + 1) = (\mathop t\nolimits^2 + t + 1 + \dfrac{1}{4} - \dfrac{1}{4}) \\
  {\text{ }} = (\mathop t\nolimits^2 + t + \mathop {(\dfrac{1}{2})}\nolimits^2 + \dfrac{3}{4}) \\
  {\text{ }} = [\mathop {(t + \dfrac{1}{2})}\nolimits^2 + \dfrac{3}{4}] \\
 $
On substituting, the integration becomes
$\mathop I\nolimits_2 = \smallint \mathop a\nolimits^2 \sqrt {\left[ {\mathop {\left( {t + \dfrac{1}{2}} \right)}\nolimits^2 + \dfrac{3}{4}} \right]} {\text{ }}dt$
$ \Rightarrow \mathop a\nolimits^2 \int {\sqrt {\left[ {\mathop {\left( {t + \dfrac{1}{2}} \right)}\nolimits^2 + \mathop {\left( {\dfrac{{\sqrt 3 }}{2}} \right)}\nolimits^2 } \right]} {\text{ }}dt} $
Using the special integration of type \[\int {\sqrt {\mathop x\nolimits^2 + \mathop b\nolimits^2 } } \]
\[\int {\sqrt {\mathop x\nolimits^2 + \mathop b\nolimits^2 } } dx = \dfrac{1}{2}x\sqrt {\mathop x\nolimits^2 + \mathop b\nolimits^2 } + \dfrac{{\mathop b\nolimits^2 }}{2}\log \left| {x + \sqrt {\mathop x\nolimits^2 + \mathop b\nolimits^2 } } \right| + C\]
On comparing : $x = t + \dfrac{1}{2};{\text{ }}b = \dfrac{{\sqrt 3 }}{2}$
Hence, \[\mathop I\nolimits_2 = \dfrac{{\mathop a\nolimits^2 }}{2}\left( {t + \dfrac{1}{2}} \right)\sqrt {\mathop {\left( {t + \dfrac{1}{2}} \right)}\nolimits^2 + \dfrac{3}{4}} + \dfrac{{\mathop a\nolimits^2 3}}{8}\log \left| {\left( {t + \dfrac{1}{2}} \right) + \sqrt {\mathop {\left( {t + \dfrac{1}{2}} \right)}\nolimits^2 + \dfrac{3}{4}} } \right| + C\]
Simplifying the expressions under the square-roots
\[
   \Rightarrow \dfrac{{\mathop a\nolimits^2 }}{2}\left( {t + \dfrac{1}{2}} \right)\sqrt {(\mathop t\nolimits^2 + t + 1)} + \dfrac{{\mathop a\nolimits^2 3}}{8}\log \left| {\left( {t + \dfrac{1}{2}} \right) + \sqrt {(\mathop t\nolimits^2 + t + 1)} } \right| + \mathop c\nolimits_2 \\
   \Rightarrow \dfrac{{\mathop a\nolimits^2 }}{4}\left( {2t + 1} \right)\sqrt {(\mathop t\nolimits^2 + t + 1)} + \dfrac{{\mathop a\nolimits^2 3}}{8}\log \left| {\left( {t + \dfrac{1}{2}} \right) + \sqrt {(\mathop t\nolimits^2 + t + 1)} } \right| + \mathop c\nolimits_2 \\
 \]
Now, substituting back (t, x,) what we have let so far.
We know, $t = \tan \theta $
           \[\tan \theta = \dfrac{{\sqrt {ax - \mathop a\nolimits^2 } }}{a}{\text{ }}\]
Hence, \[t = \dfrac{{\sqrt {ax - \mathop a\nolimits^2 } }}{a}{\text{ }}\]
 \[(\mathop t\nolimits^2 + t + 1) = \mathop {\left( {\dfrac{{\sqrt {ax - \mathop a\nolimits^2 } }}{a}} \right)}\nolimits^2 + \dfrac{{\sqrt {ax - \mathop a\nolimits^2 } }}{a} + 1\]
Simplify taking LCM ${a^2}$
\[
   = \dfrac{{ax - \mathop a\nolimits^2 + a\sqrt {ax - \mathop a\nolimits^2 } + \mathop a\nolimits^2 }}{{\mathop a\nolimits^2 }} \\
   = \dfrac{{ax + a\sqrt {ax - \mathop a\nolimits^2 } }}{{\mathop a\nolimits^2 }} \\
 \]
On substituting values of $t$ and \[(\mathop t\nolimits^2 + t + 1)\] integration $\mathop I\nolimits_2 $,
\[{I_2} = \dfrac{{{a^2}}}{4}\left[ {\left( {\dfrac{{2\sqrt {ax - {a^2}} }}{a} + 1} \right)\sqrt {\dfrac{{ax + a\sqrt {ax - {a^2}} }}{{{a^2}}}} } \right] + \dfrac{{3{a^3}}}{8}\log \left[ {\dfrac{{\sqrt {ax - {a^2}} }}{a} + \dfrac{1}{2} + \sqrt {\dfrac{{ax + a\sqrt {ax - {a^2}} }}{{{a^2}}}} } \right] + {c_2}\]
Cancel the common terms of numerator and denominator, in the second term the denominator 2a.
\[ \Rightarrow \dfrac{{{{{a^2}}}}}{4}\left[ {\left( {\dfrac{{2\sqrt {ax - {a^2}} + a}}{{{a}}}} \right)\dfrac{{\sqrt {ax + a\sqrt {ax - {a^2}} } }}{{{a}}}} \right] + \dfrac{{3{a^3}}}{8}\log \left[ {\dfrac{{2\sqrt {ax - {a^2}} }}{{2a}} + \dfrac{a}{{2a}} + \dfrac{{2\sqrt {ax + a\sqrt {ax - {a^2}} } }}{{2a}}} \right] + {c_2}\]
Using the property of logarithm:
$\log \dfrac{a}{b} = \log a - \log b$
\[ \Rightarrow \left[ {\dfrac{{2\sqrt {ax - {a^2}} + a}}{4}\sqrt {ax + a\sqrt {ax - {a^2}} } } \right] + \dfrac{{3{a^2}}}{8}\log \left[ {2\sqrt {ax - {a^2}} + a + 2\sqrt {ax + a\sqrt {ax - {a^2}} } } \right] - \dfrac{{3{a^2}}}{8}\log (2a) + {c_2}\]
The term \[\left( { - \dfrac{{3{a^2}}}{8}\log (2a)} \right)\] is constant, therefore combine it with integration constant ${c_2}$.
\[ \Rightarrow \dfrac{{2\sqrt {ax - {a^2}} + a}}{4}\sqrt {ax + a\sqrt {ax - {a^2}} } + \dfrac{{3{a^2}}}{8}\log \left[ {2\sqrt {ax - {a^2}} + a + 2\sqrt {ax + a\sqrt {ax - {a^2}} } } \right] + \mathop c\nolimits_3 \]
Where $\mathop c\nolimits_3 $ is the integration of constants
We know, $I = \mathop I\nolimits_1 + \mathop I\nolimits_2 $
\[\smallint \sqrt {ax + a\sqrt {ax - \mathop a\nolimits^2 } } dx\]
\[ = \dfrac{2}{{3a}}\mathop {\left( {ax + a\sqrt {ax - \mathop a\nolimits^2 } } \right)}\nolimits^{\dfrac{3}{2}} + \mathop c\nolimits_1 \] \[ + \dfrac{{2\sqrt {ax - {a^2}} + a}}{4}\sqrt {ax + a\sqrt {ax - {a^2}} } + \dfrac{{3{a^2}}}{8}\log \left[ {2\sqrt {ax - {a^2}} + a + 2\sqrt {ax + a\sqrt {ax - {a^2}} } } \right] + \mathop c\nolimits_3 \]
\[ = \dfrac{2}{{3a}}\mathop {\left( {ax + a\sqrt {ax - \mathop a\nolimits^2 } } \right)}\nolimits^{\dfrac{3}{2}} \; + \dfrac{{2\sqrt {ax - {a^2}} + a}}{4}\sqrt {ax + a\sqrt {ax - {a^2}} } + \dfrac{{3{a^2}}}{8}\log \left[ {2\sqrt {ax - {a^2}} + a + 2\sqrt {ax + a\sqrt {ax - {a^2}} } } \right] + c\]
Where $c = {c_1} + {c_3}$
On comparing with the given solution:
\[\smallint \sqrt {ax + a\sqrt {ax - \mathop a\nolimits^2 } } dx,0 < a < 2 = \dfrac{2}{{3a}}\mathop {\{ ax + a\sqrt {ax - \mathop a\nolimits^2 } \} }\nolimits^{\dfrac{3}{2}} - \dfrac{{\sqrt a }}{2}\left| {A + B} \right| + c.\]
For the values of A and B options (B) and (C) is correct.

Final answer: For the values of A and B options (B) and (C) are correct.

Note: Carefully do the calculation, emphasize on each and every step. Especially while substituting the values of ${I_1}$ and ${I_2}$
For instance, while using the special integration of particular functions of type \[\int {\sqrt {\mathop x\nolimits^2 + \mathop b\nolimits^2 } } \],
Compare with the integral judiciously, $\mathop {\left( x \right)}\nolimits^2 $ implies the whole square of variable function and
\[\mathop {\left( b \right)}\nolimits^2 \] implies the whole square of constant function.