Note: I must agree with vv that this seems to be a case of using a generic formula.
So I converted this into a comment.
Here is a simplified version, where I set t = 0 and a=b=2 right away.

restart;
G:=n->sum(cos(2*Pi/n*j)^2*sin(2*Pi/n*j)^2,j=0..n-1); #The simplified version
Ga:=n->add(cos(2*Pi/n*j)^2*sin(2*Pi/n*j)^2,j=0..n-1); #Version using add
H:=n->sum(combine(cos(2*Pi/n*j)^2*sin(2*Pi/n*j)^2),j=0..n-1); #Version combining before summing
G(n); #Wrong for some n
H(n); #Seems to be better
combine~([seq(G(n),n=1..5)]);
combine~([seq(Ga(n),n=1..5)]);
[seq(H(n),n=1..5)];

##Test of n/8 as the answer:
simplify(combine~([seq(G(n)-n/8,n=1..25)]));
simplify(combine~([seq(Ga(n)-n/8,n=1..25)]));
simplify([seq(H(n)-n/8,n=1..25)]);

@Carl Love His expected output can be obtained by doing

PDEtools:-declare(phi(X));
Diff(conjugate(phi(X)),x1);
                             

I checked that mgear is still present in Maple 7 by doing
interface(verboseproc=2);
eval(`dsolve/numeric/mgear`);

I looked at the help page for dsolve/numeric/mgear in Maple 7 via What's New. There it says:

"The mgear method is obsolete, and is no longer available in Maple. The rosenbrock and lsode methods are avaialble for the numeric solutions of stiff initial value problems."

@olivertwist My code works in Maple 2016 (but it doesn't use Douglas Meade's Shoot package, so you don't need the with(Shoot) line).

Which Maple version do you have?
The eval[recurse] command is only available in more recent versions. If the result of that command (i.e. sys) is not free of variables with p added to them (as in fp) then you must have an older version.
In older versions you can do the following (which also works in more recent versions):

SBS:=fppp(eta) = diff(fpp(eta), eta), fpp(eta) = diff(fp(eta), eta), fp(eta) = diff(f(eta), eta),gp(eta) = diff(g(eta), eta), mp(eta) = diff(m(eta), eta), np(eta) = diff(n(eta), eta); #The order is important!!!
sys:=subs(SBS,sys1); #The first substitution is applied first, etc.

#####
I'm puzzled by your remark "I solved the problem using desolve without converting the system into first order one (RKF45)".
If you used dsolve with boundary conditions given at two points (here eta=0, eta=blt=1) then you were not using RKF45 as that method is for initial value problems only.

@olivertwist The best way to find out why something is needed is to try with and without.
In this case (for me at least) I had to isolate the derivatives for shoot to work, which meant passing from ODE1 to ODE2.
If I'm right, this simply means that the shoot procedure was written under the assumption that the derivatives are isolated on the left.

@Carl Love Yes, certainly. I just tried your code in Maple 8. It worked.
The copyright is
op(3,eval(`dsolve/numeric/bvp`));
    `Copyright (c) 2000 by Waterloo Maple Inc. All rights reserved.`
## I have also tested the code in Maple 6 and 7. `dsolve/numeric/bvp` doesn't exist in Maple 6, but does in Maple 7, and your code works in Maple 7, but not in Maple 6.

@tomleslie Yes, I missed that one.

@Doug Meade I have been doing what you describe at the end. I use dsolve with the parameters option and fsolve.
For the system in this question your shoot was roughly fifty times faster than mine. I attribute that to my use of fsolve.
Note: I just added LSSolve as a solver. That makes my procedure go about as fast as yours at least in this example.

@vv I found the horses in the subfolder 'images', so I needed:
X:=Read(cat(kernelopts(mapledir),"/data/images/fjords.jpg")):

@Carl Love Yes, I can see that I (among other people) made a comment to that question. I still have a copy.

@John Fredsted Your method works for me too. In fact it worked when I opened the next to the last section only (by left clicking on the overlapping triangles).

@vv Thanks. My (embarrassing) mistake.

@vv I may not understand what you are saying, but both limits below exist and are equal:

eval(x-a+y-2, [x=a+rho*cos(theta), y=b+rho*sin(theta)]);
limit(%, rho=0);
# and
limit(x-a+y-2,{x=a,y=b});

The latter (described in ?limit/multi) should be quite equivalent to the former in my understanding of limit.

# What confuses me in your answer is the reference to "the previous limit". What was that previous one?

 # Note: I made an embarrassing mistake. See examples by vv below.

@tomleslie I'm guessing that once the system is written as an obviously linear system fsolve will use that fact, so that there is no need for LinearSolve.
As a rather trivial example take sys and sys2 below, where sys2 is linear but sys is not. Assuming that A[i]<>0 for all i the two are equivalent, but fsolve has more trouble with sys:
restart;
sys:={seq(1/A[i]*add(j*A[j]+i*j,j=1..i)=0,i=1..3)};
sys2:={seq(add(j*A[j]+i*j,j=1..i)=0,i=1..3)};
debug(fsolve);
fsolve(sys);
fsolve(sys2);
##### Timing comparisons:
restart;
sys:={seq(1/A[i]*add(j*A[j]+i*j,j=1..i)=0,i=1..500)}:
CodeTools:-Usage(fsolve(sys)):
memory used=1.23GiB, alloc change=0.61GiB, cpu time=15.61s, real time=14.81s, gc time=11.19s
restart;
sys2:={seq(add(j*A[j]+i*j,j=1..i)=0,i=1..500)}:
CodeTools:-Usage(fsolve(sys2)):
memory used=256.07MiB, alloc change=26.71MiB, cpu time=2.39s, real time=2.41s, gc time=171.88ms
##############
To Jens I should add that when fsolve turns unevaluated (as it did for you) this doesn't mean that it hasn't performed any work. It means that it wasn't able to find a solution.

@torabi I can only report what I find. I shall leave it to you to compare the results to the results reported in the paper where you found the graphs.