\(\left( P \right):\,\,y = \frac{1}{2}{x^2}\).

TXĐ : \(D = R\). Ta có \(y' = x\).

Giả sử \(A\left( {{x_1};\frac{1}{2}x_1^2} \right);\,\,B\left( {{x_2};\frac{1}{2}x_2^2} \right) \in \left( P \right)\,\,\left( {{x_1} \ne {x_2}} \right)\).

Phương trình tiếp tuyến tại điểm A của \(\left( P \right)\) là \(y = {x_1}\left( {x - {x_1}} \right) + \frac{1}{2}x_1^2 \Leftrightarrow y = {x_1}x - \frac{1}{2}x_1^2\,\,\left( {{d_1}} \right)\)

Phương trình tiếp tuyến tại điểm B của \(\left( P \right)\) là \(y = {x_2}\left( {x - {x_2}} \right) + \frac{1}{2}x_2^2 \Leftrightarrow y = {x_2}x - \frac{1}{2}x_2^2\,\,\left( {{d_2}} \right)\)

Do \(\left( {{d_1}} \right) \bot \left( {{d_2}} \right)\) nên ta có \({x_1}{x_2} =  - 1 \Leftrightarrow {x_2} = \frac{{ - 1}}{{{x_1}}}\).

Phương trình đường thẳng AB :

\(\begin{array}{l}\frac{{x - {x_1}}}{{{x_2} - {x_1}}} = \frac{{y - \frac{1}{2}x_1^2}}{{\frac{1}{2}x_2^2 - \frac{1}{2}x_1^2}} \Leftrightarrow \frac{1}{2}\left( {x - {x_1}} \right)\left( {x_2^2 - x_1^2} \right) = \left( {y - \frac{1}{2}x_1^2} \right)\left( {{x_2} - {x_1}} \right)\\ \Leftrightarrow \left( {x - {x_1}} \right)\left( {{x_2} + {x_1}} \right) = 2y - x_1^2 \Leftrightarrow \left( {{x_1} + {x_2}} \right)x - 2y - {x_1}{x_2} = 0\\ \Leftrightarrow y = \frac{1}{2}\left[ {\left( {{x_1} + {x_2}} \right)x - {x_1}{x_2}} \right] = \frac{1}{2}\left[ {\left( {{x_1} + {x_2}} \right)x + 1} \right]\end{array}\)

Do đó diện tích hình phẳng giới hạn bởi \(AB,\,\,\left( P \right)\) là :

\(\begin{array}{l}S = \frac{1}{2}\int\limits_{{x_1}}^{{x_2}} {\left( {\left( {{x_1} + {x_2}} \right)x + 1 - {x^2}} \right)dx} \\ \Leftrightarrow \frac{9}{4} = \frac{1}{2}\left. {\left( {\left( {{x_1} + {x_2}} \right)\frac{{{x^2}}}{2} + x - \frac{{{x^3}}}{3}} \right)} \right|_{{x_1}}^{{x_2}}\\ \Leftrightarrow \frac{9}{4} = \frac{1}{2}\left[ {\left( {{x_1} + {x_2}} \right)\left( {\frac{{x_2^2}}{2} - \frac{{x_1^2}}{2}} \right) + \left( {{x_2} - {x_1}} \right) - \frac{{x_2^3 - x_1^3}}{3}} \right]\\ \Leftrightarrow \frac{9}{2} = \frac{1}{2}\left( {{x_1} + {x_2}} \right)\left( {x_2^2 - x_1^2} \right) + \left( {{x_2} - {x_1}} \right) - \frac{{x_2^3 - x_1^3}}{3}\\ \Leftrightarrow 27 = 3\left( {{x_1}x_2^2 - x_1^3 + x_2^3 - x_1^2{x_2}} \right) + 6\left( {{x_2} - {x_1}} \right) - 2x_2^3 + 2x_1^3\\ \Leftrightarrow 27 = 3{x_1}x_2^2 - 3x_1^2{x_2} + x_2^3 - x_1^3 + 6\left( {{x_2} - {x_1}} \right)\\ \Leftrightarrow 27 =  - 3\left( {{x_2} - {x_1}} \right) + \left( {{x_2} - {x_1}} \right)\left( {x_1^2 + x_2^2 - 1} \right) + 6\left( {{x_2} - {x_1}} \right)\\ \Leftrightarrow 27 = 3\left( {{x_2} - {x_1}} \right) + \left( {{x_2} - {x_1}} \right)\left( {x_1^2 + x_2^2 - 1} \right)\\ \Leftrightarrow 27 = \left( {{x_2} - {x_1}} \right)\left( {x_1^2 + x_2^2 + 2} \right)\\ \Leftrightarrow 27 = \left( {{x_2} - {x_1}} \right)\left( {x_1^2 + x_2^2 - 2{x_1}{x_2}} \right)\\ \Leftrightarrow 27 = \left( {{x_2} - {x_1}} \right){\left( {{x_2} - {x_1}} \right)^2} = {\left( {{x_2} - {x_1}} \right)^3}\\ \Leftrightarrow {x_2} - {x_1} = 3\end{array}\)

Thay \({x_2} = \frac{{ - 1}}{{{x_1}}}\) ta có : \(\frac{{ - 1}}{{{x_1}}} - {x_1} = 3 \Leftrightarrow  - 1 - x_1^2 - 3{x_1} = 0 \Leftrightarrow \left[ \begin{array}{l}{x_1} = \frac{{ - 3 - \sqrt 5 }}{2} \Rightarrow {x_2} = \frac{2}{{3 + \sqrt 5 }}\\{x_1} = \frac{{ - 3 + \sqrt 5 }}{2} \Rightarrow {x_2} = \frac{{ - 2}}{{ - 3 + \sqrt 5 }}\end{array} \right. \Leftrightarrow {\left( {{x_1} + {x_2}} \right)^2} = 5\).

Chọn B.