• gpu timing measurements
  
• Different fft  benchmarks:
• fftw
• acml
• intel ipp sysperf, phils fftbnch
  
• AMD ACML library
• 20feb13: acmlbench timing on rserv2.
  
• FFTW Performance:

• fftw bench measurements on various computers

  cpu	date	version	serverType
  summer1	27sep09:		intel L5420 harpertown cpu
  aserv11	xx.xx	3.3.2	intel X5550 nehalem cpu fftw
  megs3	21feb13	3.3.2	intel a2600 I7 quad core sandybridge cpu .
  	21feb13	3.3.3
  rserv2	27mar13:	3.3.3	AMD 6272 bulldozer interlagos with 64 cores. openSUSE 12.2 (kernel 3.4.33, gcc 4.7.1

  
  
• 15feb13 compare rserv2 sse3,avx, and megs3 avx
• 21feb12: fftw version differences
  
• 
  
• Intel Performance Primitives (ipp) library
• 15aug11: fft times aserv11 using intel benchmark (plot of times)
• 05feb13: fft times megs3 (sandybridge:Intel(R) Core(TM) i7-2600 @3.40GHz ) using intel benchmark (plot of times)
  
• 18aug11: fft times aserv11 using  my bencmark (table of times)
• aserv11 (nehalem), adslinux(i7 2600Q sandybridge,opensuse11.2, megs3 (i7 2600Q sandybridge, openSuse 12.2)
  
• Code long pulse processing times.
• dec11     : rserv1 ( running 45 clp threads simultaneously)
• 05apr06: ntp time jumps on the wapps 2003-2006. What was causing them.
• 30mar06: Look at when ntpd on aeron lost sync 2003-2006
• 10oct05: Some linux kernels see no speed up when running 2 processes on a dual processor cpu
• 19apr05: wapp clock drifts when taking data.
• 19apr05: ntp time server drifts when wapps take data.

Different fft benchmarks:

• fftw benchmark. this is provided by fftw. I've added some scripts to drive it and generate html for the results:
• Let Dbase=/share/megs/phil/x101/fftw/fftw-3.3.2
• Let Drserv2=Dbase/archsrc/rserv2/
  
• bench - their script to run their benchmark
• Drserv2/bench  .. their bench script location
  
• Dbase/tests/README : a description of bench options.s
• Dbase/tests/bench.c      : source for executable
  
• benchphil :  my routine that calls bench for various fftlengths
• Drserv2/tests : location
  
• call: benchphil nthreads  .. setting nthreads > 1 will perform ffts with nthreads per fft
• sends output to bench.out (this is later parsed to generate the html. This file gets appended to so you should delete it before starting a set of measurements.
• benchphilloop: script that  has a bunch of calls to benchphil. Normaly each line is a different set of threads
• Drserv2/tests : location
• benchhtml.pl: generate an html table from the results in bench.out. this can have multiple runs of benchphil
• Drserv2/tests : location
• sends output to bench.html
• it looks in Drserv2/config.log (output from config) to see whether avx or sse was used in the build.
• The fftw timing tables on this page have been pasted here from bench.html
• intel performance primitive (intel IPP) benchmark
• It is installed in: /share/phil/intel/ .. ./ipp, /lib
  
• perfsys : a bench mark provided by intel. (an example of measurements)
  
• ps_ipps: intel script to drive benchmark
• /share/phil/intel/ipp/tools/intel64/perfsys/ps_ipps
• before using:  source /share/phil/intel/ipp/bin/ippvars.csh intel64
• sets environment variables
• sysperf.sc: script (written by me) to drive perfsys
• location: /share/megs/phil/x101/intel/tests
• sends output to /tmp/ipps/
• sysperf.pro: idl routine to plot results in /tmp/ipp..
  
• fftbnch: fft benchmark using ipp lib written by me
• /share/megs/phil/x101/intel/tests location
• ffbnch.c: source
• loops doing user specified fft not in place. datainput=0.
• Can specify the number of threads and the thread affinity.
• It also queries the cpu to see what options are available (asvx,sse,etc.).
  
• fftbnch.sc: script to drive
• AMD acml math library benchmark
  
• Installation, use  info.
• acmlbnch: fft benchmark written by me using acml lib.
• Path: /share/megs/phil/x101/amd/philtest/{rserv1,rserv2}/
• acmlbnch.c : 64K complex fft, 10000 times,  in place.
• Output to screen times.
  

ACML performance:

• 64k length, in place, search for best fftformat. Use random data, reload input each fft.
• The table below shows measurements made with this routine:
• 
  

• cpu	len	mem align	-O3	-march= bdver1	-mavx	-mprefer-avx128	per loop usecs	per fft usecs	Date	lib
  rserv2	64k	-	x				1373	1290	20feb13	_fma4
  	64k	32	x				1135	1052
  	64k	-		x		x	2183	1310
  	64k	32		x		x	2142	1057
  	64k	32	x	x		x	1108	1047
  	64k	32		x	x		2133	1048
  	64k	32	x	x	x		1111	1050
  	64k	32	x	x			1114	1053

Acml notes:

• The per loop time includes the time to reload the array
• without -O3 it take about 1000 more usecs to reload the input array
• posix_memalign(32) sped things up by about 250 usecs.
• looking at the mem pointers:
• memalign(32) aligned on 64 byte boundaries
• no memalign ended up aligning on 16 bytes boundaries.
  

FFTW performance

cpu: summer1: (top)

• cpu Specs:
•  Intel(R) Xeon(R) CPU L5420  @ 2.50GHz (formerly harpertown)
  
• 6144 MB cache / quad core (12MB total).
  
• dual quad core
  
• memory: 32 GB;
  

• Software runs:
• threads of 1, 4,8  (it is an dual quad core machine)
• SSE of and on
• I think it was single precision.
  

fftw on cpu  aserv11: (top)

• cpu Specs:
•  Intel(R) Xeon(R) CPU X5550  @ 2.67GHz (formerly nehalem)
  
• 8192 KB cache / quad core
  
• dual quad core
  
• memory: 24 GB
• hyperthreading enabled.
  

• Software runs:
• threads of 1, 4,8 ,12, 16  (it is an dual quad core machine)
• SSE was used (even though the table says sse2). Since i think it was single precision (which uses sse).
  
• I think it was single precision (the binary linked to the single precision lib.

• multithreading doesn't look like it helps until you get to 16K xforms (at least for 4 threads).
• You need a long xform to use lots of threads.
• Using too many threads slows it down.
  

21feb13: fftw 3.3.2 bench mark on megs3 (sandybridge) cpu  (top)

• cpu Specs: (megs3 2012)
  
• Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (sandybridge)
  
• 8192 KB cache / quad core
  
• single quad core
  
• memory: 8 GB
• hyperthreading enabled.
• Software:
• OS was openSuse 12.2 with kernel 3.1.9-1.4. This supports the avx registers.
• fftw-3.3.2 which supports avx.
• I downloaded 3.3.2 and compiled it with avx.
  

• Software runs:
• threads of 1, 4,8
  
• bench --info-all returns:
• bench --info-all
• (name "fftw3")
• (version "fftw-3.3.2-avx")
• (cc "gcc -std=gnu99 -O3")
• I was expecting -mavx.. maybe since the lib is already compiled with that, the benchmark main program doesn't need it?
  
• (codelet-optim "")
• (benchmark-precision "single")

• Limited to 8 threads since that the max you have with 4 cores and hyperthreading..
• did better than aserv11 (nehalem without avx) by about a factor of 2 to 3.
• For the longest ffts the improvement factor decreased some. This is probably because of the cash sizes:
• aserv11 has 8mb cache/quadcore and two quadcore chips
  
• megs3 has 8mb cache/quadcore with only 1 quadcore chip.

21feb13: fftw version 3.3.3 on megs3  (top)

27mar13: fftw on rserv2 (64 core amd bulldozer)  (top)

• compare  rserv2 avx,sse2, and megs3 avx (single threaded ffts) 
• compare rserv2 avx, megs3 avx (using 4 threads for fft)
  
• table with avx  times
• table with sse2 times
  
• optimization info: amd 6200 series and it's relevance to hpc (.pdf)
  
• cpu Specs: (rserv2 mar13)
  
• AMD operteron 6272 64 cores (16 cores/chip)
  
• 2048  KB cache  
  
• memory: 128 GB
• 
  
• Software:
• OS:openSUSE 12.2 kernel 3.4.33, gcc 4.7.1
• I had previously timed things with centos6 which had  kernel 2.6.32-279.
  
• This supports the avx registers (but it is not the most recent kernel).
• The single thread times were about the same as kernel 3.4.33
  
• The multiple thread times were faster with kernel 3.4.33 (by almost a factor of 2).
  
• fftw-3.3.3 which supports avx
• I downloaded and built this on the amd machine using gcc version 4.7.1
  

• Software runs:
• threads of 1, 4,8,16
• I tried 3 configurations:
• --enable-avx .. This is the standard option for fftw. bench --info-all gave:
• (name "fftw3")
• (version "fftw-3.3.3-avx")
• cc comand used:
• (cc "gcc -std=gnu99 -O3")
• (codelet-optim "")
• (benchmark-precision "single")
•  "CFLAGS='-O3 -march=bdver1 -mprefer-avx128' . these are the optimum flags for the bulldozer/interlagos cpu
  
• bench --info-all returns:
• (name "fftw3")
• (version "fftw-3.3.3-avx")
• (cc "gcc -std=gnu99 -O3 -march=bdver1 -mprefer-avx128")
• (codelet-optim "")
• (benchmark-precision "single")
• This also had avx enabled.
• --enable-sse   this is the standard fftw option to include sse and sse2.
• bench --info-all returns:
• (name "fftw3"
• (version "fftw-3.3.3-sse2")
• (cc "gcc -std=gnu99 -O3")
• (codelet-optim "")
• (benchmark-precision "single")
  

• I got  similar timing whether i used the fftw enable avx switch, or i explicitly forced the -march=bdver1 and -mprefer-avx128"
• The table  times below are from the --enable-avx  run
• fftw uses the avx inline instructions supplied by gcc (see ./simd-support/simd-avx.h and gcc include <immintrin.h>). So the compiler may not have much flexibility when the bdver1, mprefer-avx128 options were given.
  
• This cpu has a 2.1GHz clock.
• for single threads rserv2 was 3 to 4 times slower than megs3 (I7 quad core).
• multiple threads times on rserv2  improved by about a factor of two when moving from kernel 2.2.6.32-279 to 3.4.33 
  
• I  added longer lenghths 16Mpnt,64Mpnt, 256Mpnt  each with 16 cores (after the facts i realized that the setup time took forever).
  
• rserv2 has a computation unit which has 2 cores. It  has a single floating point unit that has 2 sets of 128bit registers (1 for each core). It does not look like it can handle 256 bits avx instructions very well..
• 
  

• rserv2 (bulldozer) compare AVX,SSE2 single threaded times (and megs3 sandybridge)

  fftlen	rserv2 avx	rserv2 sse2	megs3 avx I7-2600@3.4Ghz
  1024	4.6 usecs	5.5 usecs	1.31 usecs
  4096	27.0usecs	32.5 usecs	8.4 usecs
  16384	168 usecs	167 usecs	52.4 usecs
  128k	2.48 ms	2.16 ms	.59 ms
  1024K	36.5 ms	37.31 ms	15.97 ms



• 
  

• Compare rserv2(bulldozer) and megs3(sandybridge) with 4 threads.

  fftlen	rserv2 avx	megs3 avg I7-2600@3.4GHz
  1024	36.16 usecs	12.11 usecs
  4096	47.23 usecs	14.93 usecs
  16384	97.73 usecs	25.44 usecs
  128k	710.38 usecs	189.84 usecs
  1024K	11.28 ms	5.50 ms

  
  

The table below has the avx times:   (top)

FFTW version differences

    The table below show some timing differences in different fftw versions:

notes:

• version 3.3.2, 3.3.3
• All the differences were in 8 threads. Since megs3 only has 4cores (*2 with hyperthreading) it's probably other task also running that may have made these times differ. All the single thread  times are pretty close.
  

intel performance primitives (IPP) lib: (top)

Intel ps_ipps benchmark:

15aug11 IPP ffttimes on aserv11 (.ps) (.pdf) using intel benchmark program ps_ipps.

• this is an 8 processor nehalem machine
• I tried running with 1,4,8,16 threads but the intel test program ps_ipps seemed to only use 1 thread.
• The runs with multiple threads did produce slightly different times which is why the graphs jump around a bit.
  
• ran 2^10 to 2^24 lengths
• floating point data
• in test program  used: auto, 1% timing accuracy, 10;24;+1

05feb13 IPP ffttimes on megs3 (.ps) (.pdf) using intel benchmark program ps_ipps.

• this is an 4 processor sandybridge I7 2600 @3.4ghz machine
• The os in openSuse 12.2 kernel 3.1.9-1.4. which now supports the avx registers.
  
• I tried running with 1,4,8 threads but the intel test program ps_ipps seemed to only use 1 thread.
• The runs with multiple threads did produce slightly different times which is why the graphs jump around a bit.
  
• ran 2^10 to 2^20 lengths (since only 8gb of memory)
  
• floating point data
• in test program  used: auto, 1% timing accuracy, 10;20;+1

My benchmark

18aug11: IPP table of ffttimes on aserv11/adslinux using my test program

• I wrote an fftbenchmark (x101/intel/tests/fftbnch.c) to play with the ipp fft times.
• float, complex, not in place transform.
• loop N times using same input, output array.
• run on aserv11.
• I used ippSetNumThreads() to vary the number of threads used to compute a single fft.
  
• The number of threads actually used maxed out at 2 (even though i asked for more).
• Only lenghts 8192 -> 128K allowed for 2 threads. All other lengths only used 1 thread even if i asked for more
• on they sandybridge processor, only 1 thread was ever used to compute a single fft even if i called ippSetNumThreads(n) with n > 1.
  
• the numbers in the table are probably only accurate to a few %.
• The cpus used:
• aserv11
  
• nehalem dual quad core cpus.
  
• 2.67 Ghz
• adslinux
• sandybridge quad core (DELL 8300)
  
• 3.4 GHZ
• os suse 11.4 with kernel 2.6.37.6 . linux kernel  doesn't support avx until 2.6.30.
• Threading:
  
• setting nthreads > 1 never generated a second thread?
•  This may be in the intel ipps library?
• megs3
• sandybridge quad core (DELL 8300)
  
• 3.4 GHZ
• os suse 12.2 with kernel 3.1.9-1.4 . linux kernel 
  
• Threading:
  
• setting nthreads > 1 never generated a second thread?
•  This may be in the intel ipps library?
• Looks like os change made no difference in fft times
  
• 
  

• IPP COMPLEX OUT OF PLACE FFT TIMES

  FFTlen	time usecs 1 thread			time usecs 2 threads 1 fft
  	aserv11	adslinux	megs3	aserv11	adslinux
  1K	3.4	1.4	1.5	-
  2K	7.6	3.3	3.3	-
  4K	17.3	8.4	8.3	-
  8K	37.4	18.8	18.7	26.9
  16K	89.0	44.6	43.0	55.3
  32K	193.	102.5	106.5	111.6
  64K	435.	231	227.6	234.
  128K	933.	487	483.4	496.
  256K	2296.	1153	1143.1	-
  512K	5709	3290	3227.1	-
  1024K	12391	8088	8150	-
  2048K	26776	17580	18134	-
  4096K	55970.	37458	38025	-
  8192K	129655.	83112	96197	-
  16384K	270900.	178718	213809	-

  
  

10oct05: Some linux kernels see no speed up when running 2 processes on a dual processor cpu. (top)

    The idl routine (atmclp) processes the coded long process atm data. It was used to benchmark  some of the dual processor cpus at the observatory. The data set used was:

• 100 ipps  of 10 millisecs each (so 1 second of data).
• 1 usec baud, 500 usec transmitted pulse, 5 Mhz bandwidth.
• For each ipp the tx samples were complex conjugated and then used for decoding each height.
• 601 heights for each ipp. At each height:
• The data was decoded with the xmiter samples  (2500 samples/height)
• A 4K complex transform was done and then the power was accumulated.
• So 60100 4K transforms were done as well as the decoding and power accumulation.
• This was doubled  (120200) since it was a dual beam experiment.
• The data set was read into memory. This ended up taking about 40Mb of data (all the machines had lots of memory > 1gb)
• The fftw routine was used for the fft (rather than the idl version).
• The i/o took a small fraction of the total time (< 1%). top showed 100% cpu usage while running.
• idl 6.1 was used.

For the aolc computers you should spread the jobs out over multiple cpus rather than trying to run two of the same on the same cpu (until arun gets a chance to update the kernels).
 

 home_~phil