Remember: by definition, the answer to $$ \int \sin^2 (x)\,dx $$ should be a function (more precisely the family of functions) whose derivative is $\sin^2(x)$. You had the "guess" that $$ \int \sin^2(x)\,dx = \frac 13 \frac{\sin^3(x)}{\cos(x)} + C $$ How can we check whether this works? We could take the derivative and see what we get. Of course, when we take a derivative, the $C$ goes away, since it's a constant. For the rest, we get the quotient rule: $$ \frac{d}{dx} \left[\frac 13 \frac{\sin^3(x)}{\cos(x)} \right] = \frac 13 \frac{3 \sin^2(x) \cos(x) + \sin^4(x)}{\cos^2(x)} $$ Does this match the thing we were integrating? I think it's safe to say that it doesn't.

What you were attempting to do is effectively a $u$-substitution, i.e. a "backwards chain rule". In general, $u$-substitution is more subtle than what you tried here, but what you've done looks a lot like a commonly used "trick". Here's an example where an approach like yours could work:

To calculate $\int (2x + 1)^6\,dx$, we could note that we have "stuff" to the power of $6$, i.e. $u^6$ where $u = 2x + 1$. We already know that $$ \int u^6 du = \frac 17 u^7 + C $$ so, to account for the chain rule, we might try subbing in $u = 2x+1$ and dividing by $\frac {du}{dx} = 2$. Indeed, we find that $$ \int (2x + 1)^6 du = \frac 17 (2x + 1)^7 \cdot \frac 12 + C $$ is correct! You might try differentiating to check that this is true.

So what was different here? In this case, $\frac {du}{dx}$ (the derivative of $u$, i.e. the derivative of the inside function) was a constant, so we didn't need to worry about complications from the quotient rule. In particular: $$ \frac{d}{dx} \left[\frac 17 u^7 \cdot \frac 1{(du/dx)} \right] = \frac 17 (7u^6 (du/dx))\cdot \frac 1{(du/dx)} = u^6 \frac{du/dx}{du/dx} = u^6 = (2x+1)^6 $$ I hope that problem makes a bit more sense now.

If you're in doubt you can always just differentiate the answer you get. In this case the product rule and chain rule imply that $$ \frac{d}{dx}\left(\frac{1}{3}\sin^3x\,\frac{1}{\cos x}\right) =(\cos x\sin^2x)\left(\frac{1}{\cos x}\right)+\color{blue}{\frac13\,\sin^3x\,\left(\frac{\sin x}{\cos^2 x}\right)} =\sin^2x+\frac{1}{3}\frac{\sin^4 x}{\cos^2 x}. $$ The problem is the term coloured in blue.

The chain rule doesn't work like that in reverse - in fact, if you differentiate your "result" you don't get the same function back.

The way to integrate $\sin^2 x$ is to use the power reduction identity

$$ \sin^2 x = \frac{1 - \cos 2x}{2} $$

and move from there.