@verdin OK, I will do the calculations and post it as an Answer to close this thread.

@verdin Have a look at this modified worksheet where I have picked arbitrary parameter values and attempted to find the system's equilibria.  No equilibria are found.  That's because the parameter values have been picked blindly.  With your insight into the origins of the problem you should be able to come up with a meaningful set of parameters which might yield an equilibrium.

mw.mw

@tomleslie The OP did not ask for a path integral.  He wants to integrate the given expression over (p,q,w) ∈ R³×R³×R³, where p=⟨p₁,p₂,p₂⟩, etc., are independent variables. Thus, we have a nine-fold integral over R⁹.  I haven't attempted doing that but just from the looks of it I expect it to be complicated.

@tdavid I have fixed a few errors in your worksheet.  Here is what we get now.

Download mw.mw

@verdin It appears that the steady-states of your system are expressed through the roots of a complicated transcendental equation.  I am afraid that there are no analytic expressions for the roots.  You may have to analyze stability for specific numerical choices of the problem's parameters.

@tdavid It's difficult to diagnose a problem without seeing the worksheet.  Upload it here.

To do that, in the window where you reply to this post click on the big fat green arrow.

It's not clear to me what you are asking.

Let's look at the first term.  The exponent of the cosine is 2*beta. The coefficient is a*lambda^2/2.  If we equate these we get

2*beta = a*lambda^2/2.

Is that really what you are asking?  If not, then say clearly what you want.

@vv You are quite right.  I didn't look closely enough.  Thanks for pointing this out.

@verdin The original eq[2] was undefined at (0,0,0).  The modified one is well-defined at (0,0,0), however it is not differentiable there, and therefore the equation cannot be linearized.  Here are the details.

Download mw.mw

In your worksheet you have eq[1], eq[3], eq[3].  I assume that the middle one is meant to be eq[2].

Let's evaluate your eq[2] at (x1,x2,x3)=(0,0,0):
eval(eq[2], [x1(t)=0, x2(t)=0, x3(t)=0]);

We see that Maple complains with:
Error, numeric exception: division by zero

This says that your eq[2] is undefined at (x1,x2,x3)=(0,0,0), therefore linearizing about (x1,x2,x3)=(0,0,0) is not meaningful. You need to think that over.

@Carl Love As the parameter y changes, the graph of the function makes a transition from having two extrema (the blue curve below), to having a flat plateau (red curve) to monotone decreasing (green curve).  At the plateau both the first and second derivatives are zero.

@jan123 The labels (x and y in this case) are taken from the names of the variables in the equations in sys, but you may override them by specifying labels explicitly.  For instance,

There is no direct way of specifying label positions; they are placed automatically by Maple. For full control, you may disable the labels altogether, as in:
p1 := DEtools:-DEplot(sys, [x(t),y(t)], t=-2..2, [ic], x=-1..1, y=-1..1,
    linecolor=blue, color=red, thickness=1, tickmarks=[0,0],
    labels=["",""]);

and then design your own labels with the help of plots:-textplot(), as in

p2 := plots:-textplot([[0.9,-0.1,e__1], [0.1,0.9,e__2]]);

and then display p1 and p2 together:

plots:-display([p1,p2]);

@Carl Love I also see the behavior described by the OP.  Perhaps that's specific to Linux.  Here's what I see:

Download mw.mw

@Alger

Download orthogonality.mw

@Kitonum That's a nice animation—the red segments stand out and nicely demonstrate their equal lengths and orthogonality.

It is interesting that when one of the sides of the quadrilateral degenerates to zero length, that is, when the quadrilateral turns into a triangle (even a right triangle!), the theorem's assertion still remains valid and nontrivial.  I don't recall having seen that special case stated anywhere.  One would think that Euclid would have noticed that.