[MITgcm-support] package checkpointing issue

[Matthew Mazloff]

Hi,

Just to wrap up, my performance was returned to the level it was at  
prior to the addition of the seaice package by implementation of the  
code change below.  The slowdown of ~4.5 times was do almost entirely  
to excessive pickup writing.

Thanks for the help,
-Matt





On Sep 18, 2006, at 11:13 AM, Matthew Mazloff wrote:

> Hi Martin,
>
> As you suggested, I switched
>
>       CALL PACKAGES_WRITE_PICKUP(
>      I               .FALSE., myTime, myIter, myThid )
>       CALL WRITE_CHECKPOINT(
>      I               .FALSE., myTime, myIter, myThid )
>
> with a
>
>        CALL DO_WRITE_PICKUP(
>       I        .FALSE., myTime, myIter, myThid )
>
> in pkg/ecco/the_main_loop  and I hope this will fix my performance  
> issue.  This change should be checked in at some point...or the two  
> versions should merge
>
> -Matt
>
>
> On Sep 18, 2006, at 11:05 AM, Martin Losch wrote:
>
>> Hi Matt,
>>
>> you are right, it's an ECCO issue. That's the problem with having  
>> more than one version of a routine. pkg/ecco/the_main_loop.F is  
>> supposed to be something like model/src/the_main_loop.F with model/ 
>> src/forward_step.F being inlined. Maybe you can just copy the  
>> relevant part of forward_step.F (call do_write_pickup) into pkg/ 
>> ecco/the_main_loop.F and replace the two calles to write_pickup  
>> and packages_write_pickup.
>>
>> Martin
>> On Sep 18, 2006, at 4:55 PM, Matthew Mazloff wrote:
>>
>>> Hi Martin,
>>>
>>> do_write_pickup is called form the_model_main at the end of all  
>>> time stepping; this is fine.  Look at pkg/ecco/the_main_loop.F  
>>> (so maybe it is just an ECCO issue) and see that  
>>> package_write_pickup is called at every time step.  So it seems  
>>> that one should change the call in /ecco/the_main_loop from
>>>      CALL PACKAGES_WRITE_PICKUP(
>>>      CALL WRITE_PICKUP(
>>> to a call do_write_pickup
>>>
>>> -Matt
>>>
>>>
>>> On Sep 18, 2006, at 10:38 AM, Martin Losch wrote:
>>>
>>>> Hi Matt,
>>>> have a look at do_write_pickup, which calls write_pickup and  
>>>> also PACKAGES_WRITE_PICKUP from within the same if block (with  
>>>> the same condition as below). In older versions this was done in  
>>>> a similar so that pickup files are NOT written at every  
>>>> timesteps (others would have noticed that, too, wouldn't they).  
>>>> The slowdown of 4.5 is bound to have a different cause. Maybe  
>>>> you can send me your data.seaice and your SEAICE_OPTIONS.h and  
>>>> I'll have  a look at them.
>>>>
>>>> Martin
>>>>
>>>>
>>>> On Sep 18, 2006, at 4:29 PM, Matthew Mazloff wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> When checkpointing the model state,       SUBROUTINE  
>>>>> WRITE_CHECKPOINT, first checks to see
>>>>>        IF (
>>>>>      &     ( .NOT.modelEnd .AND.  
>>>>> (permCheckPoint.OR.tempCheckPoint) )
>>>>>      &     .OR.
>>>>>      &     ( modelEnd .AND. .NOT. 
>>>>> (permCheckPoint.OR.tempCheckPoint) )
>>>>>      &    ) THEN
>>>>>         CALL WRITE_CHECKPOINT_NOW(
>>>>>      &       permCheckPoint, myTime, myIter, myThid )
>>>>>        ENDIF
>>>>>
>>>>>
>>>>> For package checkpointing it appears this is not done; there is  
>>>>> no equivalent IF statement in packages_write_pickup.  So  
>>>>> package pickups are being written at every timestep...as more  
>>>>> packages are added this is significantly hurting performance.   
>>>>> Adding the seaice package has slowed my model down by a factor  
>>>>> of 4.5, and though I haven't done any profiling, my first guess  
>>>>> is that it is a result of this significant increase in I/O.
>>>>>
>>>>> Two possible remedies...take the call to  PACKAGE_WRITE_PICKUP  
>>>>> out of THE_MAIN_LOOP and call it with WRITE_CHECKPOINT_NOW,  or  
>>>>> add an equivalent if statement in PACKAGE_WRITE_PICKUP.
>>>>>
>>>>> Thank you,
>>>>> Matt
>>>>>
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support