[MITgcm-devel] linux_amd64_ifort+mpi_ice_nas update

Mon Oct 28 17:58:00 EDT 2013

OK. Not much incentive to test further on pleiades.
david

On 10/28/13 2:25 PM, Jean-Michel Campin wrote:
> Hi,
>
> I did few more tests (none on pleiades) with ifort using linux_amd64_ifort11:
> 1) on acesgrid (intel version 13.0.0):
>   From the recent "testreport -fast" run (from last Sat):
>    http://mitgcm.org/testing/results/2013_10/tr_acesgrid-ifc_20131026_0/summary.txt
>    http://mitgcm.org/testing/results/2013_10/rs_acesgrid-ifc_20131026_0/summary.txt
>    http://mitgcm.org/testing/results/2013_10/tr_acesgrid-iad_20131026_0/summary.txt
>   there is 1 FWD test (global_with_exf.yearly) and 2 AD experiments
>    (global_ocean.cs32x15 and lab_sea) that have problems (resiously wrong in
>    the case of AD tests).
>   Same problem without MPI, but if I switch to -O1 then all problems are gone
>    (and then results are identical to the -ieee case).
> 2) on my laptop I tried using the same optfile
>   both with ifort version 12.0.4  and with version 14.0.0
>   and all the AD tests are running fine with testreport -fast.
> 3) in addtion, on yellowstone machine, we have 1 example where with -O2
>   results are wrong but -O1 fixes it.
>
> So, from my point of view, it seems that we can keep the NOOPTFLAGS
> to -O1. Not clear if seaice_growth.F needs to be in the NOOPTFILES list,
> but I believe that when it was added there it was for a good reason.
>
> Cheers,
> Jean-Michel
>
> On Mon, Oct 28, 2013 at 11:04:00AM +0000, David Ferreira wrote:
>> Good.
>> Worth giving it a try with seaice_growth out to the NOOPTFLAGS ?
>> david
>>
>>
>> On 10/25/13 9:26 PM, Menemenlis, Dimitris (3248) wrote:
>>> Hi David, just to confirm that
>>> NOOPTFLAGS='-O1 -fp-model precise'
>>> has been running stably since morning
>>> for an llc_1080 config
>>>
>>> Dimitris Menemenlis
>>>
>>> On Oct 22, 2013, at 6:23 AM, David Ferreira wrote:
>>>
>>>> Ok, I'll check in the
>>>> NOOPTFLAGS to '-O1 -fp-model precise'
>>>>
>>>> But, I run the few seaice experiments with -O2, and there were very few differences with the -O1 case.
>>>> So maybe seaice_growth.F could be pulled out of the NOOPTFLAGS altogether
>>>> (-O2 is the highest optim on pleiades, not -O3 or -fast).
>>>>
>>>> Dimitri, worth adding this test to your hi-res llc simulations ?
>>>>
>>>>
>>>> On 10/22/13 2:22 AM, Menemenlis, Dimitris (3248) wrote:
>>>>> OK.
>>>>>
>>>>> David, will you check in change to linux_amd64_ifort+mpi_ice_nas
>>>>> or do you want me to do so?
>>>>>
>>>>> I will start using updated optfile for the hi-res llc simulations
>>>>> and report if I run into any trouble.
>>>>>
>>>>> Cheers
>>>>>
>>>>> Dimitris Menemenlis
>>>>>
>>>>> On Oct 21, 2013, at 6:15 PM, Jean-Michel Campin wrote:
>>>>>
>>>>>> Hi Dimitris,
>>>>>>
>>>>>> I would vote for the conservative approach, to just change
>>>>>> NOOPTFLAGS to '-O1 -fp-model precise'
>>>>>> and keep seaice_growth.F in the NOOPTFILES list.
>>>>>> We don't need -fPIC twice (in FFLAGS and NOOPTFLAGS), so no need
>>>>>> to keep it in NOOPTFLAGS setting.
>>>>>>
>>>>>> Just to finish with few tests with ifort (v13) on acesgrid:
>>>>>> a) As I mentionned earlier, with -fast (see e.g.:
>>>>>> http://mitgcm.org/testing/results/2013_10/tr_acesgrid-ifc_20131020_0/summary.txt
>>>>>> ) there are few "FAIL" with low level of agreement and, in addition,
>>>>>> the restart test fails for all the cubed-sphere and many Non-hydrostatic
>>>>>> experiments:
>>>>>> http://mitgcm.org/testing/results/2013_10/rs_acesgrid-ifc_20131020_0/summary.txt
>>>>>>
>>>>>> b) It get better if I change from -O2 to -O1, but still most of
>>>>>> the cubed-sphere restart tests fail.
>>>>>>
>>>>>> c) If I jsut set: FOPTIM="-O1 -align -ip -fp-model precise -xHost"
>>>>>> then the testreport output + the restart are all good, and identical
>>>>>> to the default (-ieee) results.
>>>>>>
>>>>>> d) by putting back -O2 so that:
>>>>>> FOPTIM="-O2 -align -ip -fp-model precise -xHost"
>>>>>> all the restart tests pass, little changes in testreport output
>>>>>> compared to (c) except for global_with_exf.yearly (fail @ 6)
>>>>>> and few little changes in lab_sea & seaice_obcs (but for
>>>>>> lab_sea.salt_plume and seaice_obcs the agreement with ref output
>>>>>> is even better than it was in (c)).
>>>>>>
>>>>>> e) finally, replacing "-fp-model precise" with "-fp-model source"
>>>>>> does not change anything.
>>>>>>
>>>>>> And since I think it's better to have working restart, I think I will
>>>>>> change the optfile "linux_amd64_ifort11" to what I tried in (d).
>>>>>> The problem with global_with_exf.yearly might be related to a wrong
>>>>>> compiler optimisation (vectorisation type) of one source file;
>>>>>> but it takes time to figure out which one, and this might depend on
>>>>>> the compiler & mpi version.
>>>>>>
>>>>>> Cheers,
>>>>>> Jean-Michel
>>>>>>
>>>>>> On Mon, Oct 21, 2013 at 05:37:55PM +0000, Menemenlis, Dimitris (3248) wrote:
>>>>>>> David thanks for testing.  So what next?
>>>>>>> We switch to "NOOPTFLAGS='-O1 -fPIC'" and let folks give try it in bigger configs?
>>>>>>> Or do we try to push for even more aggressive optimization?
>>>>>>>
>>>>>>> Dimitris Menemenlis
>>>>>>>
>>>>>>> On Oct 21, 2013, at 10:32 AM, David Ferreira wrote:
>>>>>>>
>>>>>>>> Jean-Michel, Dimitris,
>>>>>>>> The testreports of global_ocean.cs32x15, lab_sea, offline_exf_seaice, seaice_itd, and seaice_obcs with NOOPTFLAGS=-O1 (with -noieee) give the same results as with -O0.
>>>>>>>> So nothing special at this level.
>>>>>>>> david
>>>>>>> _______________________________________________
>>>>>>> MITgcm-devel mailing list
>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>> _______________________________________________
>>>>>> MITgcm-devel mailing list
>>>>>> MITgcm-devel at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>> _______________________________________________
>>>>> MITgcm-devel mailing list
>>>>> MITgcm-devel at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>> _______________________________________________
>>>> MITgcm-devel mailing list
>>>> MITgcm-devel at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>> _______________________________________________
>>> MITgcm-devel mailing list
>>> MITgcm-devel at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel