100,000 digits with all 64 epiphany cores.
The pi digits stop tracking the real pi digits at digit #556, I assume due to having to convert the doubles to floats to run on the ecores.
Was faster though, this took about 1.25 hours, doing 125,000 just on the ARM processors took about 4 hours.
I am not going to pursue this one anymore, but now know how to pull all 64 cores together for a common task.
High level details:
I create 8 MPI tasks, one on each of the 8 arm processor cores in the cluster (2 per board)
Each MPI task creates an 8 core workgroup on the epiphany.
Each workgroup calculates its 8 digits of pi.
When each workgroup is finished, the MPI processes return the 8 digits as a string back to MPI task rank 0.
MPI task rank 0 gathers all the values and puts them in the proper order then draws the graphics for those 64 digits.
Rinse and repeat.