Thank you for your help.

I made a typing failure in here in Matrix A, sry about it.

The Matrix A is called Werte

The Matrix B is called ABC

This is the Answer i get from Maple, 3 times PI

> Optimization[LSSolve](subs(gamma = c, [seq(seq(ABC[i, j]-Werte[i, j], i = 1 .. 3), j = 1 .. 3)]), alpha = 0 .. 2*Pi, beta = 0 .. 2*Pi, c = 0 .. 2*Pi);
[0., [c = 3.14159265358980022, alpha = 3.14159265358980022,

  beta = 3.14159265358980022, ABC[1, 1] = 0., ABC[1, 2] = 0., ABC[1, 3] = 0.,

  ABC[2, 1] = 0., ABC[2, 2] = 0., ABC[2, 3] = 0., ABC[3, 1] = 0.,

  ABC[3, 2] = 0., ABC[3, 3] = 0., Werte[1, 1] = 0., Werte[1, 2] = 0.,

  Werte[1, 3] = 0., Werte[2, 1] = 0., Werte[2, 2] = 0., Werte[2, 3] = 0.,

  Werte[3, 1] = 0., Werte[3, 2] = 0., Werte[3, 3] = 0.]]